{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Deep Autoencoding Gaussian Mixture Model for Unsupervised Anomaly Detection"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "C:\\Users\\VIPLab\\AppData\\Local\\conda\\conda\\envs\\pytorch\\lib\\site-packages\\h5py\\__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.\n",
      "  from ._conv import register_converters as _register_converters\n"
     ]
    }
   ],
   "source": [
    "import numpy as np \n",
    "import pandas as pd\n",
    "import torch\n",
    "from data_loader import *\n",
    "from main import *\n",
    "from tqdm import tqdm"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## KDD Cup 1999 Data (10% subset)\n",
    "This is the data set used for The Third International Knowledge Discovery and Data Mining Tools Competition, which was held in conjunction with KDD-99 The Fifth International Conference on Knowledge Discovery and Data Mining. The competition task was to build a network intrusion detector, a predictive model capable of distinguishing between \"bad\" connections, called intrusions or attacks, and \"good\" normal connections. This database contains a standard set of data to be audited, which includes a wide variety of intrusions simulated in a military network environment. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = pd.read_csv(\"kddcup.data_10_percent_corrected\", header=None,names=['duration', 'protocol_type', 'service', 'flag', 'src_bytes', 'dst_bytes', 'land', 'wrong_fragment', 'urgent', 'hot', 'num_failed_logins', 'logged_in', 'num_compromised', 'root_shell', 'su_attempted', 'num_root', 'num_file_creations', 'num_shells', 'num_access_files', 'num_outbound_cmds', 'is_host_login', 'is_guest_login', 'count', 'srv_count', 'serror_rate', 'srv_serror_rate', 'rerror_rate', 'srv_rerror_rate', 'same_srv_rate', 'diff_srv_rate', 'srv_diff_host_rate', 'dst_host_count', 'dst_host_srv_count', 'dst_host_same_srv_rate', 'dst_host_diff_srv_rate', 'dst_host_same_src_port_rate', 'dst_host_srv_diff_host_rate', 'dst_host_serror_rate', 'dst_host_srv_serror_rate', 'dst_host_rerror_rate', 'dst_host_srv_rerror_rate', 'type'])"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Pre-processing\n",
    "Following the paper, since the \"normal\" only comprises of approximately 20% of the entries, the \"normal\" data were considered as anomalies instead."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "data.loc[data[\"type\"] != \"normal.\", 'type'] = 0\n",
    "data.loc[data[\"type\"] == \"normal.\", 'type'] = 1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Next, the categorical variables are converted to a one hot encoding representation. My implementation is a bit different from the original paper in this aspect. Since I am only using the 10% subset to generate the columns, I get 118 features instead of 120 as reported in the paper."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>icmp</th>\n",
       "      <th>tcp</th>\n",
       "      <th>udp</th>\n",
       "      <th>IRC</th>\n",
       "      <th>X11</th>\n",
       "      <th>Z39_50</th>\n",
       "      <th>auth</th>\n",
       "      <th>bgp</th>\n",
       "      <th>courier</th>\n",
       "      <th>csnet_ns</th>\n",
       "      <th>...</th>\n",
       "      <th>dst_host_srv_count</th>\n",
       "      <th>dst_host_same_srv_rate</th>\n",
       "      <th>dst_host_diff_srv_rate</th>\n",
       "      <th>dst_host_same_src_port_rate</th>\n",
       "      <th>dst_host_srv_diff_host_rate</th>\n",
       "      <th>dst_host_serror_rate</th>\n",
       "      <th>dst_host_srv_serror_rate</th>\n",
       "      <th>dst_host_rerror_rate</th>\n",
       "      <th>dst_host_srv_rerror_rate</th>\n",
       "      <th>type</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>9</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.11</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>19</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.05</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>29</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>39</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.03</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>0</td>\n",
       "      <td>1</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>0</td>\n",
       "      <td>...</td>\n",
       "      <td>49</td>\n",
       "      <td>1.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.02</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>0.0</td>\n",
       "      <td>1</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 119 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   icmp  tcp  udp  IRC  X11  Z39_50  auth  bgp  courier  csnet_ns  ...   \\\n",
       "0     0    1    0    0    0       0     0    0        0         0  ...    \n",
       "1     0    1    0    0    0       0     0    0        0         0  ...    \n",
       "2     0    1    0    0    0       0     0    0        0         0  ...    \n",
       "3     0    1    0    0    0       0     0    0        0         0  ...    \n",
       "4     0    1    0    0    0       0     0    0        0         0  ...    \n",
       "\n",
       "   dst_host_srv_count  dst_host_same_srv_rate  dst_host_diff_srv_rate  \\\n",
       "0                   9                     1.0                     0.0   \n",
       "1                  19                     1.0                     0.0   \n",
       "2                  29                     1.0                     0.0   \n",
       "3                  39                     1.0                     0.0   \n",
       "4                  49                     1.0                     0.0   \n",
       "\n",
       "   dst_host_same_src_port_rate  dst_host_srv_diff_host_rate  \\\n",
       "0                         0.11                          0.0   \n",
       "1                         0.05                          0.0   \n",
       "2                         0.03                          0.0   \n",
       "3                         0.03                          0.0   \n",
       "4                         0.02                          0.0   \n",
       "\n",
       "   dst_host_serror_rate  dst_host_srv_serror_rate  dst_host_rerror_rate  \\\n",
       "0                   0.0                       0.0                   0.0   \n",
       "1                   0.0                       0.0                   0.0   \n",
       "2                   0.0                       0.0                   0.0   \n",
       "3                   0.0                       0.0                   0.0   \n",
       "4                   0.0                       0.0                   0.0   \n",
       "\n",
       "   dst_host_srv_rerror_rate  type  \n",
       "0                       0.0     1  \n",
       "1                       0.0     1  \n",
       "2                       0.0     1  \n",
       "3                       0.0     1  \n",
       "4                       0.0     1  \n",
       "\n",
       "[5 rows x 119 columns]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "one_hot_protocol = pd.get_dummies(data[\"protocol_type\"])\n",
    "one_hot_service = pd.get_dummies(data[\"service\"])\n",
    "one_hot_flag = pd.get_dummies(data[\"flag\"])\n",
    "\n",
    "data = data.drop(\"protocol_type\",axis=1)\n",
    "data = data.drop(\"service\",axis=1)\n",
    "data = data.drop(\"flag\",axis=1)\n",
    "    \n",
    "data = pd.concat([one_hot_protocol, one_hot_service,one_hot_flag, data],axis=1)\n",
    "data.head()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "0    396743\n",
      "1     97278\n",
      "Name: type, dtype: int64\n",
      "Anomaly Percentage 0.19691065764410826\n"
     ]
    }
   ],
   "source": [
    "proportions = data[\"type\"].value_counts()\n",
    "print(proportions)\n",
    "print(\"Anomaly Percentage\",proportions[1] / proportions.sum())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Normalize all the numeric variables."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [],
   "source": [
    "cols_to_norm = [\"duration\", \"src_bytes\", \"dst_bytes\", \"wrong_fragment\", \"urgent\", \n",
    "            \"hot\", \"num_failed_logins\", \"num_compromised\", \"num_root\", \n",
    "            \"num_file_creations\", \"num_shells\", \"num_access_files\", \"count\", \"srv_count\", \n",
    "            \"serror_rate\", \"srv_serror_rate\", \"rerror_rate\", \"srv_rerror_rate\", \"same_srv_rate\", \n",
    "            \"diff_srv_rate\", \"srv_diff_host_rate\", \"dst_host_count\", \"dst_host_srv_count\", \"dst_host_same_srv_rate\", \n",
    "            \"dst_host_diff_srv_rate\", \"dst_host_same_src_port_rate\", \"dst_host_srv_diff_host_rate\", \n",
    "            \"dst_host_serror_rate\", \"dst_host_srv_serror_rate\", \"dst_host_rerror_rate\", \"dst_host_srv_rerror_rate\" ]\n",
    "\n",
    "# data.loc[:, cols_to_norm] = (data[cols_to_norm] - data[cols_to_norm].mean()) / data[cols_to_norm].std()\n",
    "min_cols = data.loc[data[\"type\"]==0 , cols_to_norm].min()\n",
    "max_cols = data.loc[data[\"type\"]==0 , cols_to_norm].max()\n",
    "\n",
    "data.loc[:, cols_to_norm] = (data[cols_to_norm] - min_cols) / (max_cols - min_cols)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I saved the preprocessed data into a numpy file format and load it using the pytorch data loader."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "scrolled": true
   },
   "outputs": [],
   "source": [
    "np.savez_compressed(\"kdd_cup\",kdd=data.as_matrix())"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "I initially implemented this to be ran in the command line and use argparse to get the hyperparameters. To make it runnable in a jupyter notebook, I had to create a dummy class for the hyperparameters."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "class hyperparams():\n",
    "    def __init__(self, config):\n",
    "        self.__dict__.update(**config)\n",
    "defaults = {\n",
    "    'lr' : 1e-4,\n",
    "    'num_epochs' : 200,\n",
    "    'batch_size' : 1024,\n",
    "    'gmm_k' : 4,\n",
    "    'lambda_energy' : 0.1,\n",
    "    'lambda_cov_diag' : 0.005,\n",
    "    'pretrained_model' : None,\n",
    "    'mode' : 'train',\n",
    "    'use_tensorboard' : False,\n",
    "    'data_path' : 'kdd_cup.npz',\n",
    "\n",
    "    'log_path' : './dagmm/logs',\n",
    "    'model_save_path' : './dagmm/models',\n",
    "    'sample_path' : './dagmm/samples',\n",
    "    'test_sample_path' : './dagmm/test_samples',\n",
    "    'result_path' : './dagmm/results',\n",
    "\n",
    "    'log_step' : 194//4,\n",
    "    'sample_step' : 194,\n",
    "    'model_save_step' : 194,\n",
    "}"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "metadata": {
    "scrolled": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Elapsed 0:28:22.371865/0:00:00.131633 -- 0:00:00.131633 , Epoch [200/200], Iter [192/194], lr 0.0001, total_loss: 0.1894, sample_energy: 0.5388, recon_error: 0.0788, cov_diag: 11.3455\n"
     ]
    },
    {
     "data": {
      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYQAAAD8CAYAAAB3u9PLAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADl0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uIDIuMS4yLCBodHRwOi8vbWF0cGxvdGxpYi5vcmcvNQv5yAAAIABJREFUeJzsnXd8FNX2wL8nnZKE3ktQ6SWU0IsUKSrqsyD6fNL0YRefPxWsWPBZn/r0IXZAREHBiiCCgEgnQXoNECCAEBIICSFld+/vj51ddpPdZJNsdpPN/X4++ezMnTszZyZ35sw599xzRSmFRqPRaDRB/hZAo9FoNOUDrRA0Go1GA2iFoNFoNBoDrRA0Go1GA2iFoNFoNBoDrRA0Go1GA2iFoNFoNBoDrRA0Go1GA2iFoNFoNBqDEH8LUBzq1KmjYmJi/C2GJkBJSEg4o5Sq6+vz6natKWs8bdsVSiHExMQQHx/vbzE0AYqIHPHHeXW71pQ1nrZt7TLSaDQaDaAVgkaj0WgMKrxCsFgUPV5ezid/HPK3KBqNxs/kmS2YLc4ZnBNPZ/Lx6uK/H67/3xq++zPZW6JVCCpUH4I7TmfkkJlj8rcYZU5eXh7JyclkZ2f7W5QKTUREBE2aNCE0NNTfomi8TMunl9CteU0W3tfHXnbLB+s4l5XH2D4xhIV49g1stii2J6fzr/nbuLFLk7ISt9xR4RWCiL8l8B3JyclERkYSExODVKYL9yJKKVJTU0lOTqZFixb+FqdSkZ1n5khqFq0bRJbqOGaLIjjIuf2fOHeRhQnWr/mEI2edtmVmWz8WLUoxf/NRerSoTYs61Qo9R67JUioZKyoV3mVkozLM85OdnU3t2rW1MigFIkLt2rW1lVVKzl7I9bjuyr2niZnyM22e/YXh76zm8JkLDPnPKg6mZKKUIjvPXOj+eWYLh4y6s9clcflTi9n713mnOg9+uYX/LNtfYN9r3/0Dk+FC2pyUxuSFOxj05ir79qxcE3nmSy//PLOFncfTycjJ8/j6bHy58SjHz110Kku/mMefR88WqJtrsnAk9UKxz1HWBICFULlejpXtessCfQ9Lzr6/Mnhi4Xa2HTvH9w/0pXPTGmw8lEqNqmFuv/w/zte/Z3shD/nP79SLDCf1Qi7bpw6jWvil19GR1AvUj4og9UIuH6w6yJwNR+jRohabDqcBsOXIOWJqV+NsVi4No6uQke3sMs7IziM8JJhdJy4pjjs/3WRfXr77FNFVQxn1wXr6XlGbGlXD6NAomoMpmSxISKZeZLi9bt9XV/D3ns14Y+k+GteowrD29Xn22nZYlGJTUhot60VyIcfEU9/tAGDbc8OIrmp1R46fuYktR89x8N/X2K2aY2lZvL8qka82HePPZ4dSs1oYM9ce5pXFe8k1W0h69VoAHv9mG8PaN2Bou/qe/4NKSYVXCDYqgYGg0fiNU+ezefq7HSzfc9pe9vu+FDo3rcHojzYAcF1sI/59Ywce+2YbEaHB7Psrg5Pp2aRfdP+1fTojB4DksxftCmX1/hTGfLapQF2bMgDrl/yA11dyOiOHpFev5cDpTKe6HZ//tdDrufvzS+M+1iamAvDz9pMF5AI4fu4ibyzdZ1+euTaJmWuT6HdFHdYknilw7JcX7+b1W2J58tsdbDl6DoCtx87RrXlNdh5PZ+R7a+x1tyVby1/4abe9LDUzh/2nMvkmIZlvEpLtCsJiUSjg8/VJZOWaeWDQFQDsPJ7OO8sP8P4dXT3uI3FHwCgEjUZTdryxdJ+TMgB4e/l+7up/qR/mp20nWLrrrxL5389k5tA0twp5JsX+UxlF1p/64y778vd/Hi/2+byBK2UAkJVrdYF9temovezmGeuYNb47H/7ubC2Nm7m5wP7dpi13Wj+WlsU7yw+wZOdJ+7EBZqw6yOKH+9sVTPLZLC6rW71kF2MQMH0IlaITwc+cO3eO999/v9A6SUlJfPnll0UeKykpiQ4dOrjdvmrVKkaOHFlsGTVlQ2a26yi+Wz9Y77Re0s7Yr+OPcc1//yD2xV8LdBgXxSPzt5bonGVFrsnCkh0nC5QfSc1i/aHUYh/vhulrWbgl2UkZAGTmmBjwxkr7emmtAwgQhSCiXUa+wJsKQVOxyDG57vjdffK8y/Li8sPWEySlZgE4uU8qIr/uPsV9c7cUKH9necFOb09I87AD3+KFwKiAcBlVxi7CF37axe4T3nkYbbRrFMXU69q73T5lyhQOHjxI586dGTp0KABLlixBRHjmmWcYPXo0U6ZMYc+ePXTu3JmxY8dy4403cuedd3LhgjWi4n//+x99+vRxew5XpKWlMWHCBA4dOkTVqlX56KOP6NSpE7///juTJk0CrB3Fq1evJjMzk9GjR3P+/HlMJhMzZsygf//+JbwjGhtnMj2PKtK45mxW8SOXioPFC16SgFAIoD1GvuDVV19l586dbN26lYULF/LBBx+wbds2zpw5Q/fu3RkwYACvvvoqb775JosWLQIgKyuLZcuWERERwYEDB7j99tuLncht6tSpdOnShe+//54VK1YwZswYtm7dyptvvsn06dPp27cvmZmZRERE8NFHHzF8+HCefvppzGYzWVlZZXErKhU5JjM7jqf7WwxNEZi1QrBSGcMIC/uS9wVr1qzh9ttvJzg4mPr163PllVeyefNmoqKinOrl5eXx4IMPsnXrVoKDg9m/v/hm85o1a1i4cCEAgwcPJjU1lfT0dPr27cujjz7KHXfcwU033USTJk3o3r07EyZMIC8vj7/97W907tzZK9dbWTmWlkX/11cWXbGS4C6yqDxw4FQml+tOZStK9yL4FOXh18jbb79N/fr12bZtG/Hx8eTmFt/14OpcIsKUKVP45JNPuHjxIr169WLv3r0MGDCA1atX07hxY+68804+//zzYp9PYyXtQm6ZKYMfH+zLzHHdmTexF/unXV0m5/CUl290HdywbspgGkZHOJWN6NDAFyKViGW7T5X6GAGhECqffeAfIiMjyciwhgQOGDCA+fPnYzabSUlJYfXq1fTo0cOpDkB6ejoNGzYkKCiIOXPmYDYXPirVFQMGDGDu3LmANfqoTp06REVFcfDgQTp27MjkyZOJi4tj7969HDlyhHr16vHPf/6Tu+66iy1bCnbuaTzjWJp33W3XxzYCIKZ2VTo1qcGgNvXodVltt9ExVcOCAfjynz29KgfA33s2sy/f0bM5z1/XrkCdhtERVDFksFHcCChf4g1HSUAoBNB9CL6gdu3a9O3blw4dOrB+/Xo6depEbGwsgwcP5vXXX6dBgwZ06tSJkJAQYmNjefvtt7n//vuZPXs2vXr1Yv/+/VSrVngOGVc8//zzxMfH06lTJ6ZMmcLs2bMBeOedd+jQoQOxsbFUqVKFq6++mlWrVtG5c2e6dOnCwoUL7Z3O3kJERojIPhFJFJEpLraHi8h8Y/tGEYnxqgA+5GIRKSU8oUnNKsye0IMaVUOZOOAylj86gO8f6OvRvhueGkL8M1dRNcw7nu1rOjbguZHWF39YsPOrb1zfFnzwj25OZSJS4L3iqULod0Udj+VKeOYq1k4Z7FS2+GHPAiGSXr2W127uCIA3dFWA9CH4W4LKQ/6Q0jfeeMNpPTQ0lN9++82pbPv27fblV155BbDOErZz50635xk4cCADBw4EoFatWvzwww8F6rz33nsFysaOHcvYsWMLv4gSIiLBwHRgKJAMbBaRH5VSjnGSdwFnlVJXiMhtwGvA6DIRqIzIyM4jI9vEY99sc7m9S7ManDyXzV/nC+aD6tmiFhsdRhS3qh/Jla3qsvW5YcWS4cpWdYmKsKZ/OHmu4HmGtavPr8V0kVgshb8rHN1BjWtUAeDOXs15cdGlf2+Ii7fup2PjuGu2NVCialgwWblm7u7fwqmvoXntqhxJLWhxvTkqltrVrWkyYptEsy05na/+2Yt2jaL4bFwcE2YVDMCYNKQl//3tAFER1te3TWmJF3wlgWMh+FsATWWgB5ColDqklMoF5gE35KtzAzDbWF4ADJEKFvXQ8flf6fPqCpLPOidq69y0BgdevpoF9/Zhw1NDnLZ9Ni6OdVMG8+IN7gcbFofZE3rYl213L9Ih11FJzuMYlqmUIrZpjQJ13r29i9P6hH7OGXFdWQhxMbXsy5uevopNTw2hdrVLuZB2vjCc6X/v6lKmm7s2ti/nma3yRRov+uAg16/niQMuA6BHi9rWazHKvdHKAsNCoKBpp6kYLF26lMmTJzuVtWjRgu+++85PEhVKY+CYw3oykN/Bba+jlDKJSDpQG/BZaMrpjGxMZkUj4yvXm4Q6uFqeubYt037eA8DgNpcSsF3bqaE9L5CnwQc2burSmKgqzvNU2F7Cnr7wgoOkwCQ5ABZ1qb9RAV/f04vsPOfRXF0MJeHuXK4UgmNZ9fAQqoeHOLnbqoeHEOTigANa1XWKkLT1V9j6VFxZIwDVwkNY+sgAmtWqar0Wm4WgFYJBhfr+Kh1KqYAKsx0+fDjDhw/36TmL+5JywNWNz38wT+ogIhOBiQDNmjUrsENJOZOZw8A3VpGVaybp1WtZd/AMu46fJy6mJje+v465d/ekflQEaRdyqR8VTvPazn0657K8OwCtuHf6rdEFw4RtL9MgD53kESFBXMh11f9x6dlRCsJDggkPce40rhcVToOoCHtfQ35cvaRdiWV7Wdv3Cy5a9v/9vQvfbjlOy3rW0NHC+iscM8teirAs/XshMBQClSPsNCIigtTUVD0nQimwTZATERFRdOWCJANNHdabACfc1EkWkRAgGkjLVwel1EfARwBxcXFeaby/7vqLiXMS7OtfbjxqT8ls445PNjqtJ716LZk5Jm6ZsY7HhrV2ygJaYhyuxhuWu+296PiVXS3c+iJvEBXhsi/DFRZ16Sva3fsiPCS4gDvMEZu/31m+gs9i/ufTVZ38NIyuYs9gCpcUQrfmNfm/oa34e77/nQ2bMaQ7lQ0qy6uxSZMmJCcnk5KS4m9RKjS2KTRLwGagpYi0AI4DtwF/z1fnR2AssB64BVihSmGSFIf3ViQ6redXBq7YdDiNWz+0JqjzijLIh6cX/vmEHpzPdp3aIdLoXI5tEs3KfSn2sg1PDmHrsXPc+0WCy/3yc3f/FiTmS5NdHD4eE0d3h/4CG+6+5Fc+NtCe7M9VnaKahW0XpRR9rqjDsn8NcB2iaxwnIFxGRuRGPHBcKVXy9JaBbyAQGhqqp330I0afwIPAUiAY+EwptUtEXgTilVI/Ap8Cc0QkEatlcJuPZCtRegmbMvDoHPnWbVFA3mBAq7putzWIjuDb+/sQU7saXV9a5lQeeqLgW3Dm+B726/r54X5sOJTGXUbn8EFDIZRERdsmqtn5gtXF2WHqUuDS13+PFs7KwnGazuASva2t+9gsgJb1XU9AdMlhFBguo0nAHiCqqIru0N4Tja9QSi0GFucre85hORsY5UuZsnJNnL/oOj11WXJztyY8sXB7gfJQB3+5t4yjrs1qepxa2/HF3L5RNO0bRV/aaOtDKIUs1Y1opx4xtdiUlEaQwK4XhheaftpNwFChOFoIhREwncoi0gS4FngZeLQ0x6oEBoJGU4CLuWbaPbe0WAOhPOXTsXHENq1BXL4JW2y4c5U8d1179pzMYN+pDK9G/9k6dMM9yPv/44N9nSKibNijjIoh18dj4jCZCyqjz8Z351haFiLiNP2nK0oywlk8VF42heGN72J/WwjvAE8Arm0hD7GGnWqVoKkcJJ7OZM2BFLLyzHbLoCwSrjWuWYU61cO5Na4JX8cne7xfrWphPH1tW5fTYJaGoCBh8og2DGpzyb3k7rHv1KTgGAO4lA7DFuvvCe7mNK4eHkLbhp45NkriMrLtUVRa63pR1gCJpvkim0qC3xSCiIwETiulEkRkYCH1igzP0y4jTWXgfHYeq/en8NS3OzjvZgYzbxJi+DnG9I7h6/hkoqt43mdQVDRPSblv4OVO6/n7Hi6rW3hqlBs6N+bU+RzG9YnxqlxF4WnIrNM+DiGyhXF1hwZ8Ni6Oga3qlUQ0J/xpIfQFrheRa4AIIEpEvlBK/cOxkqfhedpA0AQ6k7760x5lU1bUjwonIjSYI6lZdjdH+0ZRPDuyHX/r3KhA/QX39qZuZMFQTFsHZ1k/l2EhQfz++EDCQoIIDhKqFZH3KDhICigVX1A9PISqYcEMblOPRfZBe4XvY1eqRdYTp4GBpcFvCkEp9STwJIBhITyWXxl4ijYQNJWBxJSiQyYb16jC8XMXi6znDrNFOeTGMX5F7FE6+YlzEYZp3cf664sPtfyD68ojEaHBbJ86jJDgIE6cW8uWo+eK3MdmIXhjJjRP0bmMNJoKQv40C67In665uJgtyu7m8WQwlTsupYjQT6aNEKOT+19DW3lUPyzEeheL099RWsqFQlBKrSrNGAQ9aldTGUjJyCmyTsni3S9RJTSYx4e3IUisaRw0/uPyutV5+pq2bhPjlQXlQiF4A92HoNHAqDjPR2BHugiV/OLunlwf24hDr1xLRGjJrQ3xsEO0MlOU9SQi/HPAZfYoIl8QEApB0KapRvPZuDgnX3+VQl7ozWpVZccLw2le2zlU8bJSzslrwxaRlD/Jm8Y7I4rLCn+PQ9BoNB6Q4SbPjyP5I03WTRlMZo7J5bzINs/SnAk9+WXXSWpWDXObGqEktGsUxSdj4uhbBgPmAoXyaD0FhkKQ8nlzNRpvcfjMhWLvU7NaGDWrhbncZvtGbVa7KhMHlE0Y5lVuBnRpyi8B4zLSaCoTQQIjOzUs8f46EMN/lOdbHxAKQaMJdLLyTfgSGhzE//7elU1Pu8/dXxjl+J0U8HSPqcUNnRvx7xs7+luUAgSEy0h/7WgCnQs5zqkqbE2+XmQJI1D0I+M3wkKC+O9tXYqu6AcCxkLQye00gUymoRDCjMFN5TlSRVNxCQiFoA0ETaAzd8NRAH55pL9XjqcfGY0rAkIhgE5doQlsNiVZp2WOMuL7m9aqUmj9mlVdZyadNb47oN2sGtcERh8COuxUU7aIyBvAdUAucBAYr5QqkKFMRJKADMAMmJRScd6Uo1bVMN6/oytxzWu6rfP74wOdUlV3a16ThCNnAetE7qAtBI1rAsJC0F87Gh+wDOiglOoE7MfI1OuGQUqpzt5WBmDNq39Nx4aFpjNoXrsaNapeGn/gamJ4/choXBEQCgF06gpN2aKU+lUpZQv12QB4njSolJxML3k6a4DHhl3Krml7TnSntMYVAaEQdNPW+JgJwBI32xTwq4gkGLP9uUREJopIvIjEp6QUPunNLTPWA0X3G7gjxGFuYW9OyK4JPAKiDwF0H4LGK7QSkZ0uyp9WSv0AICJPAyZgrptj9FVKnRCResAyEdmrlFqdv5KnMwEC9glverWo7XL70Hb16dLM9RzCBc/rUTVNJSUgFIL+2tF4if2F+f1FZCwwEhii3Ax8UUqdMH5Pi8h3QA+ggELwFLPl0mnCQ10b9B+P8byrQrtWNYUREC4j0GGnmrJFREYAk4HrlVJZbupUE5FI2zIwDHBlcXjMyr2n7cuX1Sl9aupQw31Uw01YqqZyExAWAog2hTVlzf+AcKxuIIANSql7RaQR8IlS6hqgPvCdsT0E+FIp9UtpTurYrEszYY2NlvWq89zIdlwX26jUx9IEHgGhELTLSFPWKKWucFN+ArjGWD4ExHrzvBEObqJ2jaJKfJx1UwaTY7IgIkxwmERHo3EkIBSCFW0iaAIPk9GH8O7tXejc1LOOY1c0qlGyCCVN5SIg+hC0gaAJVHLyLABcXreanyXRVAYCQiGADqfTBCY5Jus8COEhAfOoasoxAdHKdB+CJlCxpb2uHq6jgjRlj98Ugog0FZGVIrJHRHaJyKTSHE9bCJpAJCPbqhCiqgRQd5+m3OLPVmYC/k8ptcWI3U4QkWVKqd3FPZDOy6IJVDKy8wgOEqp4IeRUoykKv1kISqmTSqktxnIGsAdoXOLj6SgjTQCSnWchIiRIZ/TV+IRy0YcgIjFAF2BjyfbXLiNNYJJjMhOurQONj/C7QhCR6sBC4BGl1HkX24vMCqm/nTSBSk6eRUcYaXyGX1uaiIRiVQZzlVLfuqqjlPpIKRWnlIqrW7eu22NpA0ETiOSaLYRphaDxEf6MMhLgU2CPUuqtUh7LO0JpNOUMbSFofIk/W1pf4E5gsIhsNf6uKenBdB+CJhDJMZkJD9F9CBrf4LewU6XUGrT7X6MplFyzthA0viNgWpoOO9UEIjl5ug9B4zsCoqWJoHuVNQFJjklbCBrfERAtTfcpawKVXJNF9yFofEZAKATQBoKmbBGR50XkeFEBECIyQkT2iUiiiEwp7XlzTGbtMtL4jIDImKVzGWl8xNtKqTfdbRSRYGA6MBRIBjaLyI8lyc9lQ7uMNL4kYFqa0nGnGv/TA0hUSh1SSuUC84AbSnPAHJOF8NCAeUw15ZyAaGm6D0HjIx4Uke0i8pmI1HSxvTFwzGE9mVIkbATdh6DxLQGhEED3IWi8QisR2eni7wZgBnA50Bk4CfzHxf6uPk1cNk1PcnSB7kPQ+JYA6UPQI5U1XmG/UiquqEoi8jGwyMWmZKCpw3oT4ISrYyilPgI+AoiLi3PZei0WRZ5Z6T4Ejc8IDIWgfUaaMkZEGiqlThqrNwI7XVTbDLQUkRbAceA24O8lPWeu2QLgUwshLy+P5ORksrOzfXZOjfeIiIigSZMmhIaWbMrVgFAIoF1GmjLndRHpjLWpJQH3AIhII+ATpdQ1SimTiDwILAWCgc+UUrtKekK7Qgj2nUJITk4mMjKSmJgY/aFVwVBKkZqaSnJyMi1atCjRMQJCIehmqylrlFJ3uik/AVzjsL4YWOyNc5rM1s+ckCDftfDs7GytDCooIkLt2rUprE+qKALGOanDTjWBhslitRBCfGghgHbBVmRK+78LDIWg268mAPGHhaCp3ASGQkD3IWgCD7tC8LGFoKm8BERL099PmkAkz3AZhQbrFl5WDBw4kPj4eH+LUW4ICIUQJILFom0ETWBhNtp0sHYZVQrMZrO/RQiMKKPQ4CDyzFohaAKLPCPsNCTIP99tL/y0i90nznv1mO0aRTH1uvaF1rlw4QK33norycnJmM1mnn32Wfbt28dPP/3ExYsX6dOnDx9++CEiwsCBA+nSpQsJCQmkpKTw+eef88orr7Bjxw5Gjx7NtGnTSEpKYsSIEfTs2ZM///yTVq1a8fnnn1O1alWn8/76669MnTqVnJwcLr/8cmbOnEn16tVdypiQkMCjjz5KZmYmderUYdasWTRs2JCBAwfSs2dPVq5cyblz5/j000/p378/ZrOZKVOmsGrVKnJycnjggQe45557WLVqFS+88AINGzZk69at7N69m5deeom5c+fStGlT6tSpQ7du3bjxxhsZNWoUW7ZsAeDAgQPcdtttJCQkeOcfYxAQFkJosNgfHo0mULD1IVQ2l9Evv/xCo0aN2LZtGzt37mTEiBE8+OCDbN68mZ07d3Lx4kUWLbo0UDwsLIzVq1dz7733csMNNzB9+nR27tzJrFmzSE1NBWDfvn1MnDiR7du3ExUVxfvvv+90zjNnzjBt2jSWL1/Oli1biIuL46233nIpX15eHg899BALFiwgISGBCRMm8PTTT9u3m0wmNm3axDvvvMMLL7wAwKeffkp0dDSbN29m8+bNfPzxxxw+fBiATZs28fLLL7N7927i4+NZuHAhf/75J99++63dnXX55ZcTHR3N1q1bAZg5cybjxo3zzg13ICAshJDgIHuInkYTKJj87DIq6ku+rOjYsSOPPfYYkydPZuTIkfTv35+FCxfy+uuvk5WVRVpaGu3bt+e6664D4Prrr7fv1759exo2bAjAZZddxrFjx6hRowZNmzalb9++APzjH//g3Xff5bHHHrOfc8OGDezevdteJzc3l969e7uUb9++fezcuZOhQ4cCVleP7ZwAN910EwDdunUjKSkJsFof27dvZ8GCBQCkp6dz4MABwsLC6NGjh30g2Zo1a7jhhhuoUqUKgP0aAe6++25mzpzJW2+9xfz589m0aVNJb7FbAkMhBIl2GWkCDpPZ1qkcEIa8x7Rq1YqEhAQWL17Mk08+ybBhw5g+fTrx8fE0bdqU559/3im1Rnh4OABBQUH2Zdu6yWQCCsbn519XSjF06FC++uqrIuVTStG+fXvWr1/vcrtNhuDgYPv5lVK89957DB8+3KnuqlWrqFatmtOx3XHzzTfzwgsvMHjwYLp160bt2rWLlLW4BERLCwsJKjcuo/PZeeSYzBxNzSLxdAbnsnIB6z861+Qso+2fr5TCYlFsOJTK/lMZJBxJ43x2Htl5ZrLziu5oyt+IlFKczsh2ue/uE+c5fu4iy3efcirPNVlYvOMkmTkm8swWsnJNHl3vgVMZTF+Z6FFdTfGwWQiVbRzCiRMnqFq1Kv/4xz947LHH7H7zOnXqkJmZaf/KLg5Hjx61v8C/+uor+vXr57S9V69erF27lsREa1vOyspi//79Lo/VunVrUlJS7MfLy8tj167CM5QMHz6cGTNmkJeXB8D+/fu5cOFCgXr9+vXjp59+Ijs7m8zMTH7++Wf7toiICIYPH859993H+PHjPbzy4hEwFoLJjxbC3r/OoxS8sXQfK/aeLta+keEhZOR49vJt0yCSR65qxZPfbqdrs5qYLIrf9xc+TL19oyh2nThPmwaR1KoWxrqDqQXq9G9Zhz8OnLGv14sM53RGDgBNa1UhSIQjqVn27XUjw6ldLYy9f2XYy95Yus9+Lhtdm9UgJCiITUlpALRrGMVVbevx7opEesTU4o5ezWgQFcH6Q6lsOpzG5XWrc22nhjSvXZXHvtnGiA4NWbTtBKfOZ5OUmkWHxlEEi7DrxHlGxTXlZPpF4prX5H8rE1n9xCC2HUtn31/nOZZ2kcvrVSP57EX2nDzP0Hb1UQq+33qC127uSKcmNTy63/7GrhAqWR/Cjh07ePzxxwkKCiI0NJQZM2bw/fff07FjR2JiYujevXuxj9m2bVtmz57NPffcQ8uWLbnvvvucttetW5dZs2Zx++23k5NjbfvTpk2jVatWBY4VFhbGggULePjhh0lPT8dkMvHII4/Qvr17F9vdd99NUlISXbt9m1z9AAAgAElEQVR2RSlF3bp1+f777wvU6969O9dffz2xsbE0b96cuLg4oqOj7dvvuOMOvv32W4YNG1bse+AJUpFSPsTFxSlXMcP//DyeY2lZ/PLIAI+OM2vtYRpERzB/8zFevKEDq/adZlPSWXrE1KR6RAjZeRa2HDnL8XMXqRoWzOA29Vm57zQNoiKoUTWUyIgQZq87wvFzF719iRofkPTqtS7LRSTBk/TX3sZdu/5tzynumh3PDw/0Jbapb5TYnj17aNu2rU/O5SuSkpIYOXIkO3e6SlBb/sjMzKR69epkZWUxYMAAPvroI7p27QrAm2++SXp6Oi+99JLb/V39Dz1t2361EERkBPBfrJkhP1FKvVqS44SFBNndMRaL4p4vEriQY2LdwVTu7NWcrcfOERYSRMKRswX27f/6SvvyT9tcpq5n+Z7iffUD1KwaitmiOJ9d8Ov/qrb1ChyzQVQE13duRL3IcA6ducCFHBN1qoez5ehZ/krP5oFBV/DM9wUbdGRECBkuzmFjVLcmbEs+x/5TmcW+Bk+JCA0iO698uOwCiTxz5bQQKjsTJ05k9+7dZGdnM3bsWLsyuPHGGzl48CArVqwos3P7TSF4c0LyY2lZHDpzgZgpPxfYNmfDkVLLWhSfjo3jrtmXvvDyf4Ha5Nrz4ghCgsXeSego74anhhR5nu4xtbh/bgKzxvfg1Pls4mJqOW3fdDiNtAs53PuF1ee6beowoquEFjgXwJy7enDnp9Yohddu7sjkhTsAiGtekwX39bHXT3z5ar7adJRbuze1T+WYdiGXyQu3079lHcb0jgHgjk82sDYxlcvqVGPSVS1pEBXBfXO3cEfPZry3IpEhbeqx/Xg6KRk5zL27J5k5JvpdUYfHvtnGkp1/8eIN7WnfKJqbZ6wrcN0L7+tDt+Y1yTVZCAkSJs3fytB29bmqbT2qhoXYZe3YOJodx9MB+PKfPakSGsyp89mM6NDQ5T0o79gGplW2TmVvExMTU2Lr4MYbb7SHh9p47bXXCnQOe5Mvv/zSZfl3331XZue04U8LwT4hOYCI2CYkL7ZCaFU/ku3J6R7X3/H8MDo+/6tHdd8Z3RmzRfHWsv2EhQQRJNZwvP4t6zB7XRJD2zegcY0qHH7lGo6lXaRKWMH5b/94YhAbD6cV2Lbl2aF0fWmZx3K3bhDJb/83EICmtaoW2N6jhVVBLJnUn2NpWXZlAPDIVS15Z/kBZtzRlfnxx+h3RR02PjWE9QdT+VuXxlwX24iqYZeaw+PDW1M/KoKQ4CDuNF76NmpVC+PjMc7W5xd39ST57EUnubY8OxSLRRERGsytcU2pHh7Cmcwcpzoz/tHN6Tg3dW3Mqn0ppF3ItZd1a26dvtg2Ucx7t3dxeX9+eqify3Ibjw1rRcPoKoXWKU/YQql9HXaqlNIZTw188RL2JqXtAvBbH4KI3AKMUErdbazfCfRUSj2Yr95EYCJAs2bNuh05UvCLP8dk5vT5HLv7Z+7dPZm9Lom0C7l8PCYOs1LETVsOwPbnhxEVEcoj8/7k+60nOPTvawgKElbuPc34WZsBq7vnbJY1GsCdv9lbxCelYbYoel7m/RCyiojFosizWAgJCuLyp6zTChT1P0g4cpaLuWb6taxTqnOXtz6EBQnJPPbNNlY/PohmtQt+AJQFhw8fJjIyktq1a2ulUMGwTZCTkZFRYIKcitCH4NGE5J7MPRseEkzTWlXZ+NQQoquEEhEaTN8rnF8OX9zVk8/XJxEZbr3kN0bF8uzIdgQZX1+D2tSzuxzeuCWWV3/ZywkfdBrnd/tUdoKChPAgqyW1qIgvfhs2CyLQMNvnQ/Ddi7lJkyYkJyeXapIVjf+wTaFZUvypEDyekNxT6kdFuN3Wr2Udpy/I0OAgalcPd6rz7f19WL77FEPa1mNQm3qlEUXjBTo0ji66ko8QkflAa2O1BnBOKdXZRb0kIAMwA6bSWBydmtTgyavbEFWlZPPjloTQ0NAST7+oqfj4UyF4dUJybxAaHMTVHa0dkDqwQ+OIUmq0bVlE/gMU1mk1SCl1ppDtHtG2YRRtG0aV9jAajcf4TSF4e0JyjcYXiNWxfisw2N+yaDTexq/jELw5IblG4yP6A6eUUgfcbFfAryKigA+NPrAC5AuWKBNBNZriUqFGKotICuBuYEEdoNRmuhcoL3KAlsUVhcnRCXCVwOZppdQPACIyA2u49H9cHUBEGimlTohIPWAZ8JBSanVhAlWQdg1aFleUFzmgcFmaK6XqFnWACqUQCkNE4v0RMlhe5QAti7flEJEQrP1d3ZRSyR7Ufx7IVEq9WZLzGccoF/cNtCzlWQ7wjix6CKRG4zlXAXvdKQMRqSYikbZlYBhQMRLoaDRohaDRFIfbAKeE+SLSSERs/WD1gTUisg3YBPyslPrFxzJqNCUmINJfG7jsvPMD5UUO0LK4osRyKKXGuSg7AVxjLB8CYkssmWvKy30DLYsryosc4AVZAqYPQaPRaDSlQ7uMNBqNRgNohaDRaDQagwqvEERkhIjsE5FEEZnig/M1FZGVIrJHRHaJyCSjvJaILBORA8ZvTaNcRORdQ77tItLVy/IEi8ifIrLIWG8hIhsNOeaLSJhRHm6sJxrbY7wsRw0RWSAie41709uP9+Rfxv9mp4h8JSIR/rovpcGXbbu8tWvjHLptO8tR9u1aKVVh/7CmvDgIXAaEAduAdmV8zoZAV2M5EutApnbA68AUo3wK8JqxfA2wBGt2117ARi/L8yjwJbDIWP8auM1Y/gC4z1i+H/jAWL4NmO9lOWYDdxvLYVgTwPn8ngCNgcNAFYf7Mc5f96WitO3y1q512/ZPu/Z7wy/lTeoNLHVYfxJ40scy/IB11rd9QEOjrCGwz1j+ELjdob69nhfO3QT4DWtenUVGIzwDhOS/P1hzRvU2lkOMeuIlOaKMxir5yv1xTxoDx4BaxnUuAob7476U8jr82rb92a6N4+m27Yd2XdFdRrabZCPZKPMJhhnWBdgI1FdKnQQwfm35s8tSxneAJwDbhMa1saZltk2y7HguuxzG9nSjvje4DEgBZhom/idiHZjl83uilDoOvAkcBU5ivc4E/HNfSoPf2nY5aNeg27YTvmrXFV0heDTJTpmcWKQ6sBB4RCl1vrCqLspKLaOIjAROK6USPDxXWd6rEKArMEMp1QW4gNWMdkeZyWL4cm8AWgCNgGrA1YWcz29tqAj8Ipe/27Uhg27b+Q/qo3Zd0RWC1yfZ8QQRCcX60MxVSn1rFJ8SkYbG9obA6TKWsS9wvVgnZJmH1bR+B6gh1pw7+c9ll8PYHg2keUEO27GTlVIbjfUFWB8iX98TsKaXOKyUSlFK5QHfAn3wz30pDT5v2+WkXYNu267wSbuu6ArBPsmO0bt+G/BjWZ5QRAT4FNijlHrLYdOPwFhjeSxWH6ytfIwRfdALSLeZmqVBKfWkUqqJUioG63WvUErdAawEbnEjh02+W4z6XvmKUkr9BRwTEduMYkOA3fj4nhgcBXqJSFXjf2WTxef3pZT4tG2Xl3YNum27wTft2hsdL/78w9qrvx9rRMbTPjhfP6ym13Zgq/F3DVb/3G/AAeO3llFfgOmGfDuAuDKQaSCXIjEuw5pHJxH4Bgg3yiOM9URj+2VelqEzEG/cl++Bmv66J8ALwF6sieXmAOH+ui8VpW2Xx3at27bv27VOXaHRaDQaoOK7jDQajUbjJbRC0Gg0Gg2gFYJGo9FoDCrUfAh16tRRMTEx/hZDE6AkJCScUR7MO+ttdLvWlDWetu0KpRBiYmKIj4/3txiaAEVE3E10X6bodq0pazxt29plpNFoNBogQBSCUor9pzL8LYZG41WOn7vIb3tOkZ1n9rcomkqCRwpBisjLLiIDRGSLiJhE5JZ828wistX4+9Gh3GUe75Lw5aajDHt7NesPppb0EBpNueOP/SncNTues1m5/hZFU0kosg9BRIKxjrwbijU/xmYR+VEptduh2lGsubkfc3GIi0qpzi7KXwPeVkrNE5EPgLuAGcWUH4BDKRcA2HH8HL0vLw+JKj0nLy+P5ORksrOz/S1KpSEiIoImTZoQGhrqb1E8ojKNHdXPQ+kobdv2pFO5B5ColDoEICLzsGbdsysEpVSSsc3i6gD5MXJxDAb+bhTNBp6nhAqhRhXrxZ/LyivJ7n4lOTmZyMhIYmJisN4WTVmilCI1NZXk5GRatGjhb3EKxdYcKpE+0M9DKfBG2/bEZVTa/N4RIhIvIhtE5G9GWWG5zYtNeKj1MnJNHumjckV2dja1a9fWjd9HiAi1a9fWX6DlFP08lBxvtG1PLITS5vduppQ6ISKXAStEZAfgKs+6y2OKyERgIkCzZs1cniA8JBiAXHPFUwiAbvw+pqLcbzEevcqWb6yi/H/KI6W9d55YCKXK762UOmH8HgJWYZ2J6Qzu83jn3/8jpVScUiqubl3X4yrCQqyXkZNXMRWCRuMS/V7U+BhPFEKJ87KLSE0RCTeW62Cd+GK3sn7yuMvjXWxsH1A5Jh2eV1qUUlgs5VOxms3O/1+TyeSmpjOe1iuvVDIDoVKQlJREhw4dAIiPj+fhhx/2s0RWilQIhp//QayTNu8BvlZK7RKRF0XkegAR6S4iycAo4EMR2WXs3haIF5FtWBXAqw7RSZOBR0UkEWufwqclvQiz8cRUVJeRv0lKSqJt27bcf//9dO3alTlz5tC7d2+6du3KqFGjyMzMBGDz5s306dOH2NhYevToQUZGBtnZ2YwfP56OHTvSpUsXVq5cCcCsWbO46aabGDFiBC1btuSJJ54oVIZff/3V5TljYmJ48cUX6devH9988w0DBw7kqaee4sorr+S///0vR44cYciQIXTq1IkhQ4Zw9OhRAMaNG8ejjz7KoEGDmDx5chnevbJDGwiVg7i4ON59911/iwF4mLpCKbUYWJyv7DmH5c1Y3T7591sHdHRzzENYI5hKjcViVQinz+d443B+44WfdrH7RGHT2Bafdo2imHpd+yLr7du3j5kzZ/Liiy9y0003sXz5cqpVq8Zrr73GW2+9xZQpUxg9ejTz58+ne/funD9/nipVqvDf//4XgB07drB3716GDRvG/v37Adi6dSt//vkn4eHhtG7dmoceeoimTZsWOPeZM2eYNm1agXM+95y1iUVERLBmzRoAPvjgA86dO8fvv/8OwHXXXceYMWMYO3Ysn332GQ8//DDff/89APv372f58uUEBweX/kb6gcruS/fn8/D555/z5ptvIiJ06tSJadOmMWHCBFJSUqhbty4zZ84kOjqa2NhYDh06RFBQEFlZWbRu3ZpDhw65DPtMSEhgwoQJVK1alX79+tnLV61axZtvvsmiRYvYtGkTjzzyCBcvXqRKlSrMnDmT1q1bk5WVxbhx49i7dy9t27YlKSmJ6dOnExcX59X7U6FyGbnDbCiE+CNn/SxJxaV58+b06tWLRYsWsXv3bvr27QtAbm4uvXv3Zt++fTRs2JDu3bsDEBUVBcCaNWt46KGHAGjTpg3Nmze3K4QhQ4YQHR0NQLt27Thy5IhLhbBhwwaX57QxevRop/qO6+vXr+fbb63T/955551OlsioUaMqrDJwRLuMfMuuXbt4+eWXWbt2LXXq1CEtLY2xY8e6/PCIjY3l999/Z9CgQfz0008MHz7c7RiA8ePH895773HllVfy+OOPu6zTpk0bVq9eTUhICMuXL+epp55i4cKFvP/++9SsWZPt27ezc+dOOnd2NbSr9ASEQrA4PDF/Hj1Ll2Y1/ShNyfHky6WsqFatGmDtQxg6dChfffWV0/bt27e7/GItLAImPDzcvhwcHOzWl+/unPllc7fuiKOMhdWrCFRu+8B/z8OKFSu45ZZbqFOnDgC1atVy++Fhs5oHDRrEvHnzuP/++10eMz09nXPnznHllVfaj7FkyRKX9caOHcuBAwcQEfLyrGOr1qxZw6RJkwDo0KEDnTp18u5FGwRELiOT5dJLaXtyuh8lqfj06tWLtWvXkpiYCEBWVhb79++nTZs2nDhxgs2bNwOQkZGByWRiwIABzJ07F7C6aI4ePUrr1q3dHr845/SEPn36MG/ePADmzp3rZIoHCqpSDU3zP0qpIt11tu3XX389S5YsIS0tjYSEBAYPHlziYwI8++yzDBo0iJ07d/LTTz/ZxxT4KvQ4IBSC2aIfGG9Rt25dZs2axe23306nTp3o1asXe/fuJSwsjPnz5/PQQw8RGxvL0KFDyc7O5v7778dsNtOxY0dGjx7NrFmznCyD0pzTE959911mzpxJp06dmDNnjr1PIxCwj1TWzdunDBkyhK+//prUVGtutLS0NLcfHtWrV6dHjx5MmjSJkSNHunVR1qhRg+joaHtfmO0jKj/p6ek0bmwdoztr1ix7eb9+/fj6668B2L17Nzt27Cj9hbpCKVVh/rp166Zc8e7y/ar55EWq+eRF6tstx1zWKa/s3r3b3yJUSvLf9/HjxysgD9ipjPYG1AKWAQeM35pGuQDvAonAdqCrwz5jjfoHgLGqFO362y3HVPPJi9ThlMwyuQflkfLyPMyaNUu1b99ederUSY0dO1YdPnxYDRo0SHXs2FENHjxYHTlyxF73m2++UYBatWpVoceMj49XnTp1Ur169VJTp05V7du3V0optXLlSnXttdcqpZRat26datmyperTp4965plnVPPmzZVSSmVmZqqbb75ZdezYUY0ZM0Z17dpV7d+/3+V5XN1DIF550Bb9/pIvzp+7B+ftZfvsCmHepiMu65RXyssDUNnIf99///13hTU/l6NCeB2YYixPAV4zlq8BlhiKoRewUV1SIIeM35rGck1VwnZtUwiHtEKo9JhMJnXx4kWllFKJiYmqefPmKicnx2Xd0iiEgOhUdnQZZeXqwWnlmZ49e5KT4xwePGfOHDp2dBmd7DMGDBgAYMLZjXoDMNBYno11pP1ko/xz40HbICI1RKShUXeZUioNQESWASMA173lRSCVvltZYyMrK4tBgwaRl5eHUooZM2YQFlbiGQPcEnAK4YWfdjO+b/nOYlmZ2bhxo79FKA71lVInAZRSJ0WknlHuLuFjaRNBusSqdzQVhQceeIC1a9c6lU2aNInx48eX+JiRkZE+mWY1MBSCUoSFBFXIbKeaCom7hI8eJ4L0JGljZUx/HQhMnz7d3yKUmICIMrJYFCFBFde81l+AvqUY9/uU4QrC+D1tlLtL+OhxIkjlQdLGyop+HkpOae9dQCgEk0URXEGH+UdERJCamqofAh+hlHUSkYiICE+q/4g1agicEzD+CIwRK72AdMO1tBQYZiR1rAkMM8pKKXNpj1Bx0M9DySlm23ZJQLiMLBZFUJBwbceG7DuV4W9xikWTJk1ITk4mJSXF36JUGmzTDDpy++23A7TBOqFfMjAVeBX4WkTuwjpN7Cij+mKskUaJQBYwHkAplSYiL2HNEAzwoq2DuSRUxlxG+nkoHa7adnEICIVgVorgICE0WMirYBlPQ0NDy/1UjpWBr776innz5m1XSuXPFjYkf10juugBV8dRSn0GfOZd6SrP17J+HvxLQLiMzBYMhaA7ljWBg80+0N4Tja8IEIVgIVhERxppAopK6DHS+JkAUQhWCyEkSJwS3Wk0gYBu0RpfERAKwaIUQUEQHBRknyxHo6no6JHKGl8TEArBbFGEBAUREqwtBE3gofsQNL7CI4UgIiNEZJ+IJIrIFBfbB4jIFhExicgtDuWdRWS9iOwSke0iMtph2ywROSwiW42/Ek8BZLYogsTqNtKpsDWBwqWRyrpNa3xDkWGnIhIMTAeGYh2JuVlEflRK7XaodhQYBzyWb/csYIxS6oCINAISRGSpUuqcsf1xpdSC0l6E2WINOw0WwWTRncqawEA7jDS+xpNxCD2ARKXUIQARmYc126NdISilkoxtTm9jpdR+h+UTInIaqAucw4uYlSJIhOAgwaI8n51Io6kIaJeRxld44jLySgZHEekBhAEHHYpfNlxJb4uIy2m2RGSiiMSLSLy70YsWiyIkWOz5jLTbSBMI6G8aja/xRCF4nMHR7QGsicHmAOOVUjYr4kmsqQK6Y51QZLKrfT1JAmbLZRQcLPZ1jSZQ0BaCxld4ohA8zuDoChGJAn4GnlFKbbCVK6VOGpP55AAzsbqmSoQ17FTsCe60haAJDKztWXcqa3yFJwphM9BSRFqISBhwG9Zsj0Vi1P8O6+xS3+TbZksrLMDfgJ3FEdwRs81CCNIWgiZw0C4jja8pUiEopUzAg1jT+O4BvlZK7RKRF0XkegAR6W5kiBwFfCgiu4zdbwUGAONchJfOFZEdwA6gDjCtpBdhNrKd5hhpKy7kmEp6KI2m3KFdRhpf4VG2U6XUYqwpfx3LnnNY3ozVlZR/vy+AL9wcc3CxJC0Es8U6Y5ot0+nO4+k0qlHFW4fXaPyCNhA0viYwRiob6a+valsf0LlfNIGBDp3W+JqAUAgWi3UcQtWwYAAu5pr9LJFG4z20y0jjKwJCIdgshCo2hZCnFYKm4qPtA42vCQiFYDJbFULVUGuXiLYQNIGEDjvV+IqAUAgWZQ07jQizXo62EDSBgD25ndYHGh8REArBltwuLDiIINEWgiYw0H3KGl8TEArBoiAoSBARqoQGawtBE1BoA0HjKwJCIVgnyLF+TlUJCyFLWwiaAEDPmKbxNQGjEIIM+7peZDh/pV/0s0QajfdQuhNB4yMCRiEEG1dSPyqclMwc/wqk0XgD+4xpGo1vCAyFYIxDAKgaFkJ2np41TVPx0Q4jja8JCIVgsVxSCBGhwTrKSBNQaI+RxlcEhEKwTZADUCUsiGwdZaQJAHQuI42vCQiFYDHSXwM67FQTgGgTQeMbAkIhmJWDhWAoBB2ZofEWIpIkIjuM+TzijbJaIrJMRA4YvzWNchGRd0Uk0ZgvvGuJz2v86qas8RWBoRAsyj6fcnhoMEphnyxHo/ESg5RSnZVSccb6FOA3pVRL4DdjHeBqoKXxNxGYUdITao+RxtcEjkJwsBAA3Y+gKWtuAGYby7OxTgNrK//cmC98A1DDNl1sSdEGgsZXeKQQRGSEiOwzzOApLrYPEJEtImISkVvybRtrmNUHRGSsQ3k3wwxPNEzsEn8POYad6hTYmjJAAb+KSIKITDTK6iulTgIYv/WM8sbAMYd9k42yYqNHKmt8TZEKQUSCgelYTeF2wO0i0i5ftaPAOODLfPvWAqYCPYEewFSbrxWrKT2RS+b1iJJcgFIKpbCPVLZZCJUh9NRiUbyzfD/nsnILbMsxmckx+f4enD6fzdebjxVdsYw5dT6blXtPe+twfZVSXbE+Aw+IyIBC6rp6ixf4yBeRiSISLyLxKSkphZ5c9yFofIUnFkIPIFEpdUgplQvMw2oW21FKJSmltgP5HffDgWVKqTSl1FlgGTDCMKGjlFLrlbX393MumdzFQim4LrYRbRpEAtZxCAC7TpwvyeFKRXaeuUhFlJ6Vx9kLBV/gJeH3Aym8s/wAz/6wq8C2Li8uo/MLy7xynuJw1+x4nli4ndMZ2YXWS7uQy76/MspMjls+WMf4WZu9ciyl1Anj9zTwHdZn4pTNFWT82rRPMtDUYfcmwAkXx/xIKRWnlIqrW7euy/NeSn+tNYLGN3iiEEpjArvbt7GxXJJjOhEUJLx3exeu7mh109pcRg999WdJDlcqev77N9o+94t9fUFCMjuPpzvViX3xV7q8VPoXdXaemeSz1pxNF3NNBbZn5ZpL5TYzmS3klqBjPiUjx9i/8JfYiHdWM/yd1SWSzROOpVnvTWlfpiJSTUQibcvAMGAn8CNgc4GOBX4wln8ExhjRRr2AdJtrqdjnLpXkGk3x8UQheGQCF3Nfj49ZHNMaLrmM/EH6xTwAMrLzsFgUj32zjZHvrSmTc42fuZlnv98JFO5SyK+QAKavTCRmys+YzO5f+DfPWEerZ5YUKkPahVzy8h1DPMy/czrDN/mmLKX/uK4PrBGRbcAm4Gel1C/Aq8BQETkADDXWARYDh4BE4GPg/tIKoO0Dja/wRCF4ZAIXc99kY7nIY3piWjtSNzLcQ9HKjo7P/8qXm446lWXnmZmycLt9vbRuo/WHUu3LR9KyyMwpaCUAfP/n8QJl01cmWmUqxALYllxQkeSn60vLeGT+Vuux8sws2XHS3pdTXtwcllLKYbhKY42/9kqpl43yVKXUEKVUS+M3zShXSqkHlFKXK6U6KqXiS3xybSJofIwnCmEz0FJEWohIGHAbVrPYE5YCw0SkptGZPAxYapjQGSLSy4guGsMlk7tUtKhTja7NahDk54fpGePrHSBmys/cMyeBeQ6drd5wG9lIPJ1Jh6lLWbrrLwBW7D1l32Z28UK8NOCp5C9Ls/Hp/fN2qzfk34v3cN/cLRw/Z3XVHD97kae/21GoFeILSqsQygMBcAmaCkKRCkEpZQIexPpy3wN8rZTaJSIvisj1ACLSXUSSgVHAhyKyy9g3DXgJq1LZDLxo+5IC7gM+wWpaHwQK908Ugytb1cOiKODOKEtcRfo48vv+ot1dpeWeOQkknbnAhFmXPkpnrk0i8XQGH/5+kM/WHHaqv3TXKbq8+CvJZ7PsZduOnfNoDEf+e3v8rPMcFI8t2MbcjUfZdDiN0pKRnUfMlJ+JmfKzXRF5SkV+mdrCTpV2Gml8RIgnlZRSi7H6Rh3LnnNY3oyzC8ix3mfAZy7K44EOxRHWU2xfhd/9eZxb45oWUbtozmTmEBURyp6T54ltWoNV+07Tqn4kjWpUsdfxlU+8KFx9Ed88Y729f2NCvxZcMCKhHvtmGwCz1yXx9LXtOJOZww3T1zKyU9HjqEwOL+ZvtyQXGFVrS0Ee5AVT7VDKBfty+sU8alUL83jfI6lZtKxX3Sty+Bo9UlnjawJipHJ+bP0Is9cllfpYS3acJG7aclo9s4Qbpq9l8Y6TjJu5mT6vrnAKm/R27L3Zokg4Um2RCqMAABVYSURBVPyva1cf0DZl4A6b398Wt59w5Kx9m62/IT95Dv0Pj369rUBmTpsFEeyFF3GEQ6BAUYc7lpZld1sBDH9nNe+tcH0NFQZtIGh8REAqhL/3aAZYxyKUJoVFSkYO983d4lR2v8O6Y9jkJ/ncMZ5gc4O0fmYJF/J1Cr+34gA3z1hfbKVQEp/5h6sPccVTi3l8gbXT29Ed9MbSfU51z2Xl8u2WZP6zzLk8/3vaFrJaGn1wPjuPo6lZTm6ioga09399JX1fXeFUtv7QmZIL4UfsfT1+lUJTmQhIheDoHpjsENlTXLq/vNwb4hRJjslC+6lLncpsA+tSiumKKmknqqMLqLDxB//39TYe/XobX2xwjqLK/562KZV5m47R+5XfmL4ykW+3JLM5yXMFd9P76xjwxkq+3HTkUqGLy0vNzOE/v+5z279guyU7j6eTkV24tVSe0PMhaHxNQCoEgFHdrF0aP2z1NEL2EluOnuWHrQXDNV3hzfQQMVN+5v++tvr1bS+34KDi/YuK6nR9dcneIo+Rm6/D2GxRfPLHIZ76bge/uUkHEVTAZWSV45uEZE6mZ/PG0n08+vU2Rn2w3qnez9tPuu38TzydCeCkfFwpvGd/2Ml7KxJZk+jeEjCZLYx8bw3//LzkUaD+oiJ3jGsqFgGrEAa0KnrMgitOn8/mpvfXMWneVo/q3/T+uhKdxx0LtySTmWOyvyRDDGvHbFH2tBgr97nP0VPUy+OD3w8WKUP+OalHfbCOaT/v4cuNR93sUbwO0D0nL6UVeeDLLXz8xyGP9z12Novhb6/mWJo1MurXXX+xeIc13DavEMvm+v+tBWCjF6KefIU2EDS+JmAVwnWxjezLnsbbn0y/SI9//1as85RFzqQOU5fyxwHr167N/XX5U4vtaTEOnHKfA+hoWpbbbSVly9FzRdYpTjho/n6JY2lZ9hc8WK0ud530n/xxmH2nMuj/+krSL+YxcU6CfZs7d9nGw2nsNpRQRfza1mGnGl8RsAoBYHAba0bilIwc0rNc+46/jj/GrLWHiZnyM71fWVFg+w2dLymW/dOu9ui8vS6rZV/uHlOzkJpFIzgrtN/3pzB73RG39e/P1wnuK85kej7yekU+t9NXm47R//WV9vX3fkvkCTd9Pz9uu+QCXH8w1WmbF9JUlCv0jGkaX+PROISKyt+6NGbF3tP2r/4/nhhErWphVAu/dNlPLHDf6dy/ZR1evrEj/ze0NUFBEBYSxIJ7e/PKkr1OoZkxU3522u+qtvWpXS2cyIgQckwWNiedJa55TeId9vEUs0U5Jakb+9mmYh/DFySU4NpcceLcxWJ1PDviiSXojTBYX6FdRhpfE9AK4cqWzv0Itq/Q0XFNee2WTkXuf//AK6geHkJ1BwUSF1OLXpfVKvQFmGOyMP0O61S657Pz6NQkmrG9Y7jsKevYviWT+gNw9X//KFKGPLOFz9e7twgCiUfm/cn3xQgCOJbPPeaJhVCB9IEdbSBofEVAu4yiq4YSFlLwEufHH+PWD9ez5kDh8emu9gW498rLXZbbvj6b1Lw0gjkqIpTxfVs4hcK2bRhF24ZRRcoP1mgdTyKDAoHiKAOAlxfvcVp/4Mui3WV5RaTlLl9UQO2lqdAEtEIA6BFTy2X5psNp/OPTjS63PT68NQBNHV7sjkRGhLJ/2tU0io4ocK7f/u9Krnfo0C6MOXf1KHCM4e3rO63vPll01lFNYFNeMsdqAp+AVwiv3tzR47q//msAu14Yzv0DL2fH88OoFxXhtm5YSBDrnhzCHT2b2cvG9G7O5XWrezygqH/LuvxraCv7+t6XRhSwHKavLDpMVOM5HRtH+1sEj/F0bgmNxlsEdB8CQJOaVTn8yjWs2pfCwZRMpv3s7Gbo2qyGPayyVf1Ie3lkRKhHx3/5xo7MNeLzbbO2uWP5owM46JCoDeCmrk1ISr3A+L4tiAgNJlj3JJYp/pxAqbjY2oIl0MKnNOWWgLcQwJoCYFCbetzd/zKev66d07Z5E3sD0LOFa9eSJ9zeo5lTv4E7rqgXyfD2DZzKgoOEx4e3oU51a0K+4gyoa1GnWvEEzceTV7cp1f4VEZPFv/MzFIeQYKtCqFj9HpqKTKVQCI6M69vCvrznxRGEhQSxbeowPr+rR4mP+cpNHVkzebA3xCO2aQ0W3Nubf13Vqsi6H4+Jo35UOD1cKLOHBl9RpHK5x03neCBT3PkU/ElosPXxrEhKTFOxqXQKwZEqYVb3QXSVUMJDyo8rIS6mFpOuasn6JwtXMiKw8amr+Pqe3mx9bqjTtqY1q5aliMWmcY2iLajCeOvWWK/IYapACsGWtsSkLQSNj6iUCsFxJHF5pmF04S9Rx+iTGlUvTRozsHVdru7YoEB0SoOoCLubqFvzwkdQR0WEcM+AyzySs3/LOkXWKW3XiO1rubRUpJer7Zp9OfOfpnJTKRXCrPE92PDkEH+LUWrcfezOGt+DyIhQbunmPIldvahw7rnycn58sC+fjevutO3WOOe6ocFBHs8y1tqhMx5g8cP9+feNztFdpVEILetVL5Wrp0PjKL64qydXtqpbodwvtj6EimTVaCo2HikEERkhIvtEJFFEprjYHi4i843tG0Ukxii/Q0S2OvxZRKSzsW2VcUzbtnrevLDCiAgNpkG0+5DS8khc85pEV3GOfCoqPP2Gzo2d1m0D5zo1qVHgWC/9rQNLJvXnrn4tGNO7OV/c3dOlm+ezcXEFyvIP4GvXKKrAiOBR3Uo+lWnHJtH2l+JNXRuz6KF+7HphuMf7L3qoP/1a1uHlGzvwydjuRe9QTggxUp+btIWg8RFFhp2KSDAwHRgKJAObReRHpdRuh2p3AWeVUleIyG3Aa8BopdRcYK5xnI7AD0opx7zSdxhzK2uKYMF9fQAY/eF6ewrn4kyG8/eezbh3QMFO5IX39eFirpnwkGDaNozi2ZGXorBa14/kme93OtVv3aDgCGubayM85P/bO/cYuaoygP++e+e1u7Ov7q70sV1sLVRbXi2lD0RUHhJbA0FRKETxFYxEfJD4AF/RKKIxBE1QQVHRKIIgShoNMYioMSm2AQrYWkrpa+3aZWn31dmdx/384547Ozs7uzvtzM7MLueXbO65554757tnvr3fnHO+8x2Hp754CTDRd37z2i7u+PPuSeVbu2QeT00SmjoacrIvxZAjnHGSawk6a2xeZTqCOQTrZWSpFMX0ENYCe1R1r6omgd8AV+SVuQK4z6QfAi6WiauzNgP3lyKsBR742IbsYrjF84p/wd125Zl0tU0sf+6prVwwyRyA4wh3Xbuaxz594VhezrcayBH0EM5ePNbzeOcZ491rHYFPXrSMq9csZu9tG2mtH+uh7Lt9Ew9+bAPbvnQJ9314ordX2HXG9ofImUu4+/3nZtPzp1hEOFsJh6yXkaWyFLMwbRGQG5z+ELBusjKqmhaRfqANyA0WdDUTDcnPRCQDPAx8Qwus0ReRG4AbALq6uvIvz3me/vKlE3Yj++aVZ/LNK4tfgV0Km87yF9s9cuP5NNWFCy6ca6oL8/MPnceqxWMT1S31ER7++Pm854f+BkKOCDe/Y3n2+mcveyO3PvIcD9ywPpvXHo+yuqsle/61y1fy1UdfoD4SYthsDpQbaPCylfPZctMFqMKHfv6vMj1x7RAsohseLd+ufBbLVBRjEApNB+a/uKcsIyLrgOOqmjv+cJ2qdotII75BeD/wiwkfonoPcA/AmjVrXnN959aGyPSFKsCqLv9lP9kez29bPnEKKNeTKX+C+tp1XVy7bqKBD+VsGXr1eYvpPpbgpouWcdcTewDf+ymXYPhosuGzYMHfbMR1hMZoiIFZtA+0ZXZTzJDRISB3RrATyA9LmS0jIiGgGcgdEL6GvOEiVe02x0Hg1/hDU5YaJ3ivN0RcNq/1X+hvXz796upiw07nbiEdC7vcuvFNNERDXLrCD/p36Yr5Be8rNPH6vWvO4ZEbzy+u4hqlLR6ZEO7EYpkpijEI/wJOE5ElIhLBf7k/mlfmUeB6k74K+Esw/CMiDvBe/LkHTF5IRNpNOgy8Cxg/e2k5YdrjM9+bCH6Hr1jYxBmLmtl3+6aiJmvzh70mI7eHkMuqrlb23b6J5fMbC14v5Jl5xTmLTmiepRa58PQOtu171XoaWSrCtAZBVdPAJ4DHgJ3Ag6r6goh8XUQuN8XuBdpEZA9wM5DrmnohcEhVc3dSjwKPicgO4BmgG/hxyU/zGucfn7/ohNwxT4b2eJS7rl3Njz8w0f20EIEdKNYgBD2JYhfFBQQTrxvP9HsQt26sbpym6Vy1i2XdkjaOJzP86fmecoo3J3hydy/9CTucVk6Kinaqqn8E/piX95Wc9Ah+L6DQvX8F1uflDQPnFipvOXliFYrkGUw0F8NtV57J4zuPEAsXtwZSRHj5WxtPWKbAEeeO953DD66rbhiSIl21i+KyladwxqImbrr/abbs+C9t8SgtdWHCrjNu/UfYlazRjYQcevpHsk4AjiP0Do6yoDlG33CSgUSK0XQG1xEynvLvw4Oc3dlMezxKfyJFZ2sdo2mPhohLezzK0o44p59SfFj3cqGq2Tr39w3T2VqPp4ojwrHjSa7/6VO89fSOgp5plpNjzoe/tlSXzWu7snMNxXIyL56zOpvZtv9o1ne/ymRdtQFEJHDVPmGDEHIdfvHhddz4q+38bfcrZDzFUy1p9bLIxEWNzx48NuU9YVdoioVpjIWIhV329g7zpgWNuI6QyiiK/6IWEVyBjEIq7TEwkqIhEqIu4jI8mubFI0Oc2laPI4LrCB3xKGnP33ccfPfhnoER3ji/kV09g7x5WRuHj42w95VhmuvCE3oET+7u5ZI7nmR1VwvJtEfv0CjLOuKEXIetL/dx2usa6RtOsrtnkJULm1jUWkfYdRhIpBgYSdEej+KpHx6kb2iUV4aSxKMheodGaa4Lc3ZnC8eTaVRhJJ2hKRZGBPqGknQfS7BiYROxkEvYFRKpDN1HE7Q2RBhIpAi7Dota6+jpHyHjKS31YbbvP8qy18VJZ5ThZJo3dMR9Y+4IB/qO09EYJZHM8Nvth7jw9A7e0NFA31AS1xF2Hh5g8bx6jgyO8uzBY7x71SIaoiGiIYeh0TSbzlrAW04rPlpyIaxBmOO85bR2nuue+7uu3fvB89jbOzRunUIVKcZVu2jmNUSyYdoDPE9JZjySGY+I65D2lIyniMBoys/LqJ8XdoVY2GUgkSIadolHQ9lV60FIEEdgOJkhmfZIZTw8VVTh0NEETx84Sr95gQ4k0rw6nGQklaG5PoKqEnL83omnSkbHftmHHKExFiLsOohALOzQXNdKfTTEgb5hUhllV89AdmHjvIYIiZTvYrurZxCAHQf7GRxNAzBkjvnsOTJEfyJFMu3Rn0ixbd9RkhkPVXi+eyBbrmdgBCDbMwJfpsZYmJFUBtWJdTzX3Y8jMJIaWxgZCTkk0x5pT3nm4DHqwi6JVIbGaCgraz6O+PNvqrC/7zghRxgcTfPPl/rwTBumMv73F7h2/213L39/sXec8d7XN5yVZcuOw6Q8/zmbYiFWLmyyBsEyNb/8yEm/h2YVzXXhrGtsDTCtq3ap62scR4g5buFhwknW6BUq6+b0qOLRkD+7l8PClrqC4dVrhdxhpeDcP/rngbuz52k2nX9PPhlPcR1BjVGc7L50xsN1/F5RcC2oJ+Mpgv+lO8K4Mrl4xojn5wd1B9mTyauqeDpWR6nUxM8pi2WOMa2rtqreo6prVHVNR0dpv+pey+S/BMUMWzmOjFv7kpue7sUZGMngcya7L+Q62bzgGJR3Tf2BwZisXifnev5zBNemkldkfB2lYg2CxVJ+inHVtlhqDjtkZLGUGRO+JXDVdoGfquoLVRbLYpkWKRA+qGYRkV5g/ySX2xkfO6la1IocYGUpxFRynKqqFR+/mSV6DVaWQtSKHFAG3Z5VBmEqRGSbqha3Wuo1IAdYWWpZjmKpJXmtLLUrB5RHFjuHYLFYLBbAGgSLxWKxGOaSQbin2gIYakUOsLIUolbkKJZaktfKMpFakQPKIMucmUOwWCwWS2nMpR6CxWKxWEpg1huEcoUZPoH6FovIEyKyU0ReEJFPmfx5IvJnEXnRHFtNvojI9418O0RkdZnlcUXkaRHZYs6XiMhWI8cDZmEUIhI153vM9deXWY4WEXlIRHaZttlQxTb5jPlunheR+0UkVq12KYVK6nat6bWpw+r2eDlmXq/9mBmz8w9/0c9LwFIgAjwLrJjhOhcAq026EdgNrAC+A3zB5H8B+LZJbwT+hB/fZj2wtczy3Iy/49wWc/4gcI1J/wj4uEnfCPzIpK8BHiizHPcBHzXpCNBSjTbBDyz3MlCX0x4frFa7zBbdrjW9trpdHb2uuuKX2EgbgMdyzm8BbqmwDH/Aj3v/H2CByVsA/Mek7wY255TPlitD3Z3A48BFwBajhK8Aofz2wV81u8GkQ6aclEmOJqOskpdfjTYJIo3OM8+5BbisGu1S4nNUVberqdfm86xuV0GvZ/uQUaEww4sqVbnphq0CtgKnqOphAHMMdp2fSRnvBD4HBPsrtgHH1N/lLr+urBzmer8pXw6WAr3Az0wX/yci0kAV2kT9vbq/CxwADuM/53aq0y6lUDXdrgG9Bqvb46iUXs92gzBtmOEZq1gkDjwMfFpVB6YqWiCvZBlF5F3AEVXdXmRdM9lWIWA18ENVXQUMM34b1XxmTBYzlnsFsARYCDQA75yivqrp0DRURa5q67WRwep2/odWSK9nu0GYNszwTCAiYfx/ml+p6u9M9v9EZIG5vgA4MsMyvhm4XET2Ab/B71rfCbSISBC0MLeurBzmejPwahnkCD77kKpuNecP4f8TVbpNAC4BXlbVXlVNAb8Dzqc67VIKFdftGtFrsLpdiIro9Ww3CBUPMywiAtwL7FTVO3IuPQpcb9LX44/BBvkfMN4H64H+oKtZCqp6i6p2qurr8Z/7L6p6HfAEcNUkcgTyXWXKl+VXlKr2AAdFZLnJuhh/u8iKtonhALBeROrNdxXIUvF2KZGK6nat6DVY3Z6Eyuh1OSZeqvmHP6u/G98j44sVqO8C/K7XDuAZ87cRf3zuceBFc5xnygv+husvAc8Ba2ZAprcx5omxFHgK2AP8Foia/Jg532OuLy2zDOcA20y7/B5orVabAF8DdgHPA7/E3wesKu0yW3S7FvXa6nbl9dquVLZYLBYLMPuHjCwWi8VSJqxBsFgsFgtgDYLFYrFYDNYgWCwWiwWwBsFisVgsBmsQLBaLxQJYg2CxWCwWgzUIFovFYgHg/995Ij7ItAVnAAAAAElFTkSuQmCC\n",
      "text/plain": [
       "<matplotlib.figure.Figure at 0x28bd6a0c160>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "phi tensor([ 0.0042,  0.9909,  0.0032,  0.0017], device='cuda:0') mu tensor([[ 0.2632,  1.0881,  0.1980],\n",
      "        [-1.6156,  0.8422, -0.4742],\n",
      "        [-0.1509,  0.9353,  0.1885],\n",
      "        [ 0.2075,  0.8453,  0.2090]], device='cuda:0') cov tensor([[[ 6.7847, -1.0272,  2.1146],\n",
      "         [-1.0272,  2.6332, -0.5521],\n",
      "         [ 2.1146, -0.5521,  0.7540]],\n",
      "\n",
      "        [[ 6.0255, -0.7950,  2.0536],\n",
      "         [-0.7950,  0.2630, -0.2839],\n",
      "         [ 2.0536, -0.2839,  0.7041]],\n",
      "\n",
      "        [[ 7.2182, -0.9259,  1.9753],\n",
      "         [-0.9259,  1.7871, -0.4390],\n",
      "         [ 1.9753, -0.4390,  0.7156]],\n",
      "\n",
      "        [[ 7.2324, -1.0829,  2.2385],\n",
      "         [-1.0829,  1.6545, -0.4776],\n",
      "         [ 2.2385, -0.4776,  0.7902]]], device='cuda:0')\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|████████████████████████████████████████████████████████████████████████████████| 194/194 [00:08<00:00, 23.20it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "======================TEST MODE======================\n",
      "N: 198371\n",
      "phi :\n",
      " tensor([ 0.0007,  0.9984,  0.0006,  0.0003], device='cuda:0')\n",
      "mu :\n",
      " tensor([[-0.2610,  2.6686, -0.1194],\n",
      "        [-1.4987,  0.8442, -0.4426],\n",
      "        [-1.0019,  1.8008, -0.0609],\n",
      "        [-0.3958,  2.1473, -0.1113]], device='cuda:0')\n",
      "cov :\n",
      " tensor([[[ 4.6235,  0.3735,  1.3941],\n",
      "         [ 0.3735,  5.3684, -0.3503],\n",
      "         [ 1.3941, -0.3503,  0.5130]],\n",
      "\n",
      "        [[ 6.2791, -0.7967,  2.1476],\n",
      "         [-0.7967,  0.3265, -0.2902],\n",
      "         [ 2.1476, -0.2902,  0.7400]],\n",
      "\n",
      "        [[ 5.2532,  0.5667,  1.2209],\n",
      "         [ 0.5667,  3.7280, -0.3171],\n",
      "         [ 1.2209, -0.3171,  0.5404]],\n",
      "\n",
      "        [[ 5.4533,  0.1742,  1.6589],\n",
      "         [ 0.1742,  4.7808, -0.3665],\n",
      "         [ 1.6589, -0.3665,  0.6084]]], device='cuda:0')\n",
      "Threshold : 4.2397990226745605\n",
      "Accuracy : 0.9746, Precision : 0.9683, Recall : 0.9540, F-score : 0.9611\n"
     ]
    }
   ],
   "source": [
    "solver = main(hyperparams(defaults))\n",
    "accuracy, precision, recall, f_score = solver.test()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### I copy pasted the testing code here in the notebook so we could play around the results."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Incrementally compute for the GMM parameters across all training data for a better estimate"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "N: 198371\n",
      "phi :\n",
      " tensor([ 0.0007,  0.9984,  0.0006,  0.0003], device='cuda:0')\n",
      "mu :\n",
      " tensor([[-0.2610,  2.6686, -0.1194],\n",
      "        [-1.4987,  0.8442, -0.4426],\n",
      "        [-1.0019,  1.8008, -0.0609],\n",
      "        [-0.3958,  2.1473, -0.1113]], device='cuda:0')\n",
      "cov :\n",
      " tensor([[[ 4.6220,  0.3708,  1.3943],\n",
      "         [ 0.3708,  5.3609, -0.3502],\n",
      "         [ 1.3943, -0.3502,  0.5129]],\n",
      "\n",
      "        [[ 6.2786, -0.7967,  2.1474],\n",
      "         [-0.7967,  0.3265, -0.2902],\n",
      "         [ 2.1474, -0.2902,  0.7399]],\n",
      "\n",
      "        [[ 5.2330,  0.5535,  1.2254],\n",
      "         [ 0.5535,  3.7158, -0.3141],\n",
      "         [ 1.2254, -0.3141,  0.5394]],\n",
      "\n",
      "        [[ 5.4512,  0.1713,  1.6592],\n",
      "         [ 0.1713,  4.7742, -0.3663],\n",
      "         [ 1.6592, -0.3663,  0.6083]]], device='cuda:0')\n"
     ]
    }
   ],
   "source": [
    "solver.data_loader.dataset.mode=\"train\"\n",
    "solver.dagmm.eval()\n",
    "N = 0\n",
    "mu_sum = 0\n",
    "cov_sum = 0\n",
    "gamma_sum = 0\n",
    "\n",
    "for it, (input_data, labels) in enumerate(solver.data_loader):\n",
    "    input_data = solver.to_var(input_data)\n",
    "    enc, dec, z, gamma = solver.dagmm(input_data)\n",
    "    phi, mu, cov = solver.dagmm.compute_gmm_params(z, gamma)\n",
    "    \n",
    "    batch_gamma_sum = torch.sum(gamma, dim=0)\n",
    "    \n",
    "    gamma_sum += batch_gamma_sum\n",
    "    mu_sum += mu * batch_gamma_sum.unsqueeze(-1) # keep sums of the numerator only\n",
    "    cov_sum += cov * batch_gamma_sum.unsqueeze(-1).unsqueeze(-1) # keep sums of the numerator only\n",
    "    \n",
    "    N += input_data.size(0)\n",
    "    \n",
    "train_phi = gamma_sum / N\n",
    "train_mu = mu_sum / gamma_sum.unsqueeze(-1)\n",
    "train_cov = cov_sum / gamma_sum.unsqueeze(-1).unsqueeze(-1)\n",
    "\n",
    "print(\"N:\",N)\n",
    "print(\"phi :\\n\",train_phi)\n",
    "print(\"mu :\\n\",train_mu)\n",
    "print(\"cov :\\n\",train_cov)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "train_energy = []\n",
    "train_labels = []\n",
    "train_z = []\n",
    "for it, (input_data, labels) in enumerate(solver.data_loader):\n",
    "    input_data = solver.to_var(input_data)\n",
    "    enc, dec, z, gamma = solver.dagmm(input_data)\n",
    "    sample_energy, cov_diag = solver.dagmm.compute_energy(z, phi=train_phi, mu=train_mu, cov=train_cov, size_average=False)\n",
    "    \n",
    "    train_energy.append(sample_energy.data.cpu().numpy())\n",
    "    train_z.append(z.data.cpu().numpy())\n",
    "    train_labels.append(labels.numpy())\n",
    "\n",
    "\n",
    "train_energy = np.concatenate(train_energy,axis=0)\n",
    "train_z = np.concatenate(train_z,axis=0)\n",
    "train_labels = np.concatenate(train_labels,axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compute the energy of every sample in the test data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [],
   "source": [
    "solver.data_loader.dataset.mode=\"test\"\n",
    "test_energy = []\n",
    "test_labels = []\n",
    "test_z = []\n",
    "for it, (input_data, labels) in enumerate(solver.data_loader):\n",
    "    input_data = solver.to_var(input_data)\n",
    "    enc, dec, z, gamma = solver.dagmm(input_data)\n",
    "    sample_energy, cov_diag = solver.dagmm.compute_energy(z, size_average=False)\n",
    "    test_energy.append(sample_energy.data.cpu().numpy())\n",
    "    test_z.append(z.data.cpu().numpy())\n",
    "    test_labels.append(labels.numpy())\n",
    "\n",
    "\n",
    "test_energy = np.concatenate(test_energy,axis=0)\n",
    "test_z = np.concatenate(test_z,axis=0)\n",
    "test_labels = np.concatenate(test_labels,axis=0)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [],
   "source": [
    "combined_energy = np.concatenate([train_energy, test_energy], axis=0)\n",
    "combined_z = np.concatenate([train_z, test_z], axis=0)\n",
    "combined_labels = np.concatenate([train_labels, test_labels], axis=0)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compute for the threshold energy. Following the paper I just get the highest 20% and treat it as an anomaly. That corresponds to setting the threshold at the 80th percentile."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 29,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Threshold : 4.774040222167969\n"
     ]
    }
   ],
   "source": [
    "thresh = np.percentile(combined_energy, 100 - 20)\n",
    "print(\"Threshold :\", thresh)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [],
   "source": [
    "pred = (test_energy>thresh).astype(int)\n",
    "gt = test_labels.astype(int)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [],
   "source": [
    "from sklearn.metrics import precision_recall_fscore_support as prf, accuracy_score"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "accuracy = accuracy_score(gt,pred)\n",
    "precision, recall, f_score, support = prf(gt, pred, average='binary')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Accuracy : 0.9743, Precision : 0.9677, Recall : 0.9538, F-score : 0.9607\n"
     ]
    }
   ],
   "source": [
    "print(\"Accuracy : {:0.4f}, Precision : {:0.4f}, Recall : {:0.4f}, F-score : {:0.4f}\".format(accuracy,precision, recall, f_score))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Visualizing the z space\n",
    "It's a little different from the paper's figure but I assume that's because of the small changes in my implementation."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 44,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "application/javascript": [
       "/* Put everything inside the global mpl namespace */\n",
       "window.mpl = {};\n",
       "\n",
       "\n",
       "mpl.get_websocket_type = function() {\n",
       "    if (typeof(WebSocket) !== 'undefined') {\n",
       "        return WebSocket;\n",
       "    } else if (typeof(MozWebSocket) !== 'undefined') {\n",
       "        return MozWebSocket;\n",
       "    } else {\n",
       "        alert('Your browser does not have WebSocket support.' +\n",
       "              'Please try Chrome, Safari or Firefox ≥ 6. ' +\n",
       "              'Firefox 4 and 5 are also supported but you ' +\n",
       "              'have to enable WebSockets in about:config.');\n",
       "    };\n",
       "}\n",
       "\n",
       "mpl.figure = function(figure_id, websocket, ondownload, parent_element) {\n",
       "    this.id = figure_id;\n",
       "\n",
       "    this.ws = websocket;\n",
       "\n",
       "    this.supports_binary = (this.ws.binaryType != undefined);\n",
       "\n",
       "    if (!this.supports_binary) {\n",
       "        var warnings = document.getElementById(\"mpl-warnings\");\n",
       "        if (warnings) {\n",
       "            warnings.style.display = 'block';\n",
       "            warnings.textContent = (\n",
       "                \"This browser does not support binary websocket messages. \" +\n",
       "                    \"Performance may be slow.\");\n",
       "        }\n",
       "    }\n",
       "\n",
       "    this.imageObj = new Image();\n",
       "\n",
       "    this.context = undefined;\n",
       "    this.message = undefined;\n",
       "    this.canvas = undefined;\n",
       "    this.rubberband_canvas = undefined;\n",
       "    this.rubberband_context = undefined;\n",
       "    this.format_dropdown = undefined;\n",
       "\n",
       "    this.image_mode = 'full';\n",
       "\n",
       "    this.root = $('<div/>');\n",
       "    this._root_extra_style(this.root)\n",
       "    this.root.attr('style', 'display: inline-block');\n",
       "\n",
       "    $(parent_element).append(this.root);\n",
       "\n",
       "    this._init_header(this);\n",
       "    this._init_canvas(this);\n",
       "    this._init_toolbar(this);\n",
       "\n",
       "    var fig = this;\n",
       "\n",
       "    this.waiting = false;\n",
       "\n",
       "    this.ws.onopen =  function () {\n",
       "            fig.send_message(\"supports_binary\", {value: fig.supports_binary});\n",
       "            fig.send_message(\"send_image_mode\", {});\n",
       "            if (mpl.ratio != 1) {\n",
       "                fig.send_message(\"set_dpi_ratio\", {'dpi_ratio': mpl.ratio});\n",
       "            }\n",
       "            fig.send_message(\"refresh\", {});\n",
       "        }\n",
       "\n",
       "    this.imageObj.onload = function() {\n",
       "            if (fig.image_mode == 'full') {\n",
       "                // Full images could contain transparency (where diff images\n",
       "                // almost always do), so we need to clear the canvas so that\n",
       "                // there is no ghosting.\n",
       "                fig.context.clearRect(0, 0, fig.canvas.width, fig.canvas.height);\n",
       "            }\n",
       "            fig.context.drawImage(fig.imageObj, 0, 0);\n",
       "        };\n",
       "\n",
       "    this.imageObj.onunload = function() {\n",
       "        fig.ws.close();\n",
       "    }\n",
       "\n",
       "    this.ws.onmessage = this._make_on_message_function(this);\n",
       "\n",
       "    this.ondownload = ondownload;\n",
       "}\n",
       "\n",
       "mpl.figure.prototype._init_header = function() {\n",
       "    var titlebar = $(\n",
       "        '<div class=\"ui-dialog-titlebar ui-widget-header ui-corner-all ' +\n",
       "        'ui-helper-clearfix\"/>');\n",
       "    var titletext = $(\n",
       "        '<div class=\"ui-dialog-title\" style=\"width: 100%; ' +\n",
       "        'text-align: center; padding: 3px;\"/>');\n",
       "    titlebar.append(titletext)\n",
       "    this.root.append(titlebar);\n",
       "    this.header = titletext[0];\n",
       "}\n",
       "\n",
       "\n",
       "\n",
       "mpl.figure.prototype._canvas_extra_style = function(canvas_div) {\n",
       "\n",
       "}\n",
       "\n",
       "\n",
       "mpl.figure.prototype._root_extra_style = function(canvas_div) {\n",
       "\n",
       "}\n",
       "\n",
       "mpl.figure.prototype._init_canvas = function() {\n",
       "    var fig = this;\n",
       "\n",
       "    var canvas_div = $('<div/>');\n",
       "\n",
       "    canvas_div.attr('style', 'position: relative; clear: both; outline: 0');\n",
       "\n",
       "    function canvas_keyboard_event(event) {\n",
       "        return fig.key_event(event, event['data']);\n",
       "    }\n",
       "\n",
       "    canvas_div.keydown('key_press', canvas_keyboard_event);\n",
       "    canvas_div.keyup('key_release', canvas_keyboard_event);\n",
       "    this.canvas_div = canvas_div\n",
       "    this._canvas_extra_style(canvas_div)\n",
       "    this.root.append(canvas_div);\n",
       "\n",
       "    var canvas = $('<canvas/>');\n",
       "    canvas.addClass('mpl-canvas');\n",
       "    canvas.attr('style', \"left: 0; top: 0; z-index: 0; outline: 0\")\n",
       "\n",
       "    this.canvas = canvas[0];\n",
       "    this.context = canvas[0].getContext(\"2d\");\n",
       "\n",
       "    var backingStore = this.context.backingStorePixelRatio ||\n",
       "\tthis.context.webkitBackingStorePixelRatio ||\n",
       "\tthis.context.mozBackingStorePixelRatio ||\n",
       "\tthis.context.msBackingStorePixelRatio ||\n",
       "\tthis.context.oBackingStorePixelRatio ||\n",
       "\tthis.context.backingStorePixelRatio || 1;\n",
       "\n",
       "    mpl.ratio = (window.devicePixelRatio || 1) / backingStore;\n",
       "\n",
       "    var rubberband = $('<canvas/>');\n",
       "    rubberband.attr('style', \"position: absolute; left: 0; top: 0; z-index: 1;\")\n",
       "\n",
       "    var pass_mouse_events = true;\n",
       "\n",
       "    canvas_div.resizable({\n",
       "        start: function(event, ui) {\n",
       "            pass_mouse_events = false;\n",
       "        },\n",
       "        resize: function(event, ui) {\n",
       "            fig.request_resize(ui.size.width, ui.size.height);\n",
       "        },\n",
       "        stop: function(event, ui) {\n",
       "            pass_mouse_events = true;\n",
       "            fig.request_resize(ui.size.width, ui.size.height);\n",
       "        },\n",
       "    });\n",
       "\n",
       "    function mouse_event_fn(event) {\n",
       "        if (pass_mouse_events)\n",
       "            return fig.mouse_event(event, event['data']);\n",
       "    }\n",
       "\n",
       "    rubberband.mousedown('button_press', mouse_event_fn);\n",
       "    rubberband.mouseup('button_release', mouse_event_fn);\n",
       "    // Throttle sequential mouse events to 1 every 20ms.\n",
       "    rubberband.mousemove('motion_notify', mouse_event_fn);\n",
       "\n",
       "    rubberband.mouseenter('figure_enter', mouse_event_fn);\n",
       "    rubberband.mouseleave('figure_leave', mouse_event_fn);\n",
       "\n",
       "    canvas_div.on(\"wheel\", function (event) {\n",
       "        event = event.originalEvent;\n",
       "        event['data'] = 'scroll'\n",
       "        if (event.deltaY < 0) {\n",
       "            event.step = 1;\n",
       "        } else {\n",
       "            event.step = -1;\n",
       "        }\n",
       "        mouse_event_fn(event);\n",
       "    });\n",
       "\n",
       "    canvas_div.append(canvas);\n",
       "    canvas_div.append(rubberband);\n",
       "\n",
       "    this.rubberband = rubberband;\n",
       "    this.rubberband_canvas = rubberband[0];\n",
       "    this.rubberband_context = rubberband[0].getContext(\"2d\");\n",
       "    this.rubberband_context.strokeStyle = \"#000000\";\n",
       "\n",
       "    this._resize_canvas = function(width, height) {\n",
       "        // Keep the size of the canvas, canvas container, and rubber band\n",
       "        // canvas in synch.\n",
       "        canvas_div.css('width', width)\n",
       "        canvas_div.css('height', height)\n",
       "\n",
       "        canvas.attr('width', width * mpl.ratio);\n",
       "        canvas.attr('height', height * mpl.ratio);\n",
       "        canvas.attr('style', 'width: ' + width + 'px; height: ' + height + 'px;');\n",
       "\n",
       "        rubberband.attr('width', width);\n",
       "        rubberband.attr('height', height);\n",
       "    }\n",
       "\n",
       "    // Set the figure to an initial 600x600px, this will subsequently be updated\n",
       "    // upon first draw.\n",
       "    this._resize_canvas(600, 600);\n",
       "\n",
       "    // Disable right mouse context menu.\n",
       "    $(this.rubberband_canvas).bind(\"contextmenu\",function(e){\n",
       "        return false;\n",
       "    });\n",
       "\n",
       "    function set_focus () {\n",
       "        canvas.focus();\n",
       "        canvas_div.focus();\n",
       "    }\n",
       "\n",
       "    window.setTimeout(set_focus, 100);\n",
       "}\n",
       "\n",
       "mpl.figure.prototype._init_toolbar = function() {\n",
       "    var fig = this;\n",
       "\n",
       "    var nav_element = $('<div/>')\n",
       "    nav_element.attr('style', 'width: 100%');\n",
       "    this.root.append(nav_element);\n",
       "\n",
       "    // Define a callback function for later on.\n",
       "    function toolbar_event(event) {\n",
       "        return fig.toolbar_button_onclick(event['data']);\n",
       "    }\n",
       "    function toolbar_mouse_event(event) {\n",
       "        return fig.toolbar_button_onmouseover(event['data']);\n",
       "    }\n",
       "\n",
       "    for(var toolbar_ind in mpl.toolbar_items) {\n",
       "        var name = mpl.toolbar_items[toolbar_ind][0];\n",
       "        var tooltip = mpl.toolbar_items[toolbar_ind][1];\n",
       "        var image = mpl.toolbar_items[toolbar_ind][2];\n",
       "        var method_name = mpl.toolbar_items[toolbar_ind][3];\n",
       "\n",
       "        if (!name) {\n",
       "            // put a spacer in here.\n",
       "            continue;\n",
       "        }\n",
       "        var button = $('<button/>');\n",
       "        button.addClass('ui-button ui-widget ui-state-default ui-corner-all ' +\n",
       "                        'ui-button-icon-only');\n",
       "        button.attr('role', 'button');\n",
       "        button.attr('aria-disabled', 'false');\n",
       "        button.click(method_name, toolbar_event);\n",
       "        button.mouseover(tooltip, toolbar_mouse_event);\n",
       "\n",
       "        var icon_img = $('<span/>');\n",
       "        icon_img.addClass('ui-button-icon-primary ui-icon');\n",
       "        icon_img.addClass(image);\n",
       "        icon_img.addClass('ui-corner-all');\n",
       "\n",
       "        var tooltip_span = $('<span/>');\n",
       "        tooltip_span.addClass('ui-button-text');\n",
       "        tooltip_span.html(tooltip);\n",
       "\n",
       "        button.append(icon_img);\n",
       "        button.append(tooltip_span);\n",
       "\n",
       "        nav_element.append(button);\n",
       "    }\n",
       "\n",
       "    var fmt_picker_span = $('<span/>');\n",
       "\n",
       "    var fmt_picker = $('<select/>');\n",
       "    fmt_picker.addClass('mpl-toolbar-option ui-widget ui-widget-content');\n",
       "    fmt_picker_span.append(fmt_picker);\n",
       "    nav_element.append(fmt_picker_span);\n",
       "    this.format_dropdown = fmt_picker[0];\n",
       "\n",
       "    for (var ind in mpl.extensions) {\n",
       "        var fmt = mpl.extensions[ind];\n",
       "        var option = $(\n",
       "            '<option/>', {selected: fmt === mpl.default_extension}).html(fmt);\n",
       "        fmt_picker.append(option)\n",
       "    }\n",
       "\n",
       "    // Add hover states to the ui-buttons\n",
       "    $( \".ui-button\" ).hover(\n",
       "        function() { $(this).addClass(\"ui-state-hover\");},\n",
       "        function() { $(this).removeClass(\"ui-state-hover\");}\n",
       "    );\n",
       "\n",
       "    var status_bar = $('<span class=\"mpl-message\"/>');\n",
       "    nav_element.append(status_bar);\n",
       "    this.message = status_bar[0];\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.request_resize = function(x_pixels, y_pixels) {\n",
       "    // Request matplotlib to resize the figure. Matplotlib will then trigger a resize in the client,\n",
       "    // which will in turn request a refresh of the image.\n",
       "    this.send_message('resize', {'width': x_pixels, 'height': y_pixels});\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.send_message = function(type, properties) {\n",
       "    properties['type'] = type;\n",
       "    properties['figure_id'] = this.id;\n",
       "    this.ws.send(JSON.stringify(properties));\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.send_draw_message = function() {\n",
       "    if (!this.waiting) {\n",
       "        this.waiting = true;\n",
       "        this.ws.send(JSON.stringify({type: \"draw\", figure_id: this.id}));\n",
       "    }\n",
       "}\n",
       "\n",
       "\n",
       "mpl.figure.prototype.handle_save = function(fig, msg) {\n",
       "    var format_dropdown = fig.format_dropdown;\n",
       "    var format = format_dropdown.options[format_dropdown.selectedIndex].value;\n",
       "    fig.ondownload(fig, format);\n",
       "}\n",
       "\n",
       "\n",
       "mpl.figure.prototype.handle_resize = function(fig, msg) {\n",
       "    var size = msg['size'];\n",
       "    if (size[0] != fig.canvas.width || size[1] != fig.canvas.height) {\n",
       "        fig._resize_canvas(size[0], size[1]);\n",
       "        fig.send_message(\"refresh\", {});\n",
       "    };\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.handle_rubberband = function(fig, msg) {\n",
       "    var x0 = msg['x0'] / mpl.ratio;\n",
       "    var y0 = (fig.canvas.height - msg['y0']) / mpl.ratio;\n",
       "    var x1 = msg['x1'] / mpl.ratio;\n",
       "    var y1 = (fig.canvas.height - msg['y1']) / mpl.ratio;\n",
       "    x0 = Math.floor(x0) + 0.5;\n",
       "    y0 = Math.floor(y0) + 0.5;\n",
       "    x1 = Math.floor(x1) + 0.5;\n",
       "    y1 = Math.floor(y1) + 0.5;\n",
       "    var min_x = Math.min(x0, x1);\n",
       "    var min_y = Math.min(y0, y1);\n",
       "    var width = Math.abs(x1 - x0);\n",
       "    var height = Math.abs(y1 - y0);\n",
       "\n",
       "    fig.rubberband_context.clearRect(\n",
       "        0, 0, fig.canvas.width, fig.canvas.height);\n",
       "\n",
       "    fig.rubberband_context.strokeRect(min_x, min_y, width, height);\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.handle_figure_label = function(fig, msg) {\n",
       "    // Updates the figure title.\n",
       "    fig.header.textContent = msg['label'];\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.handle_cursor = function(fig, msg) {\n",
       "    var cursor = msg['cursor'];\n",
       "    switch(cursor)\n",
       "    {\n",
       "    case 0:\n",
       "        cursor = 'pointer';\n",
       "        break;\n",
       "    case 1:\n",
       "        cursor = 'default';\n",
       "        break;\n",
       "    case 2:\n",
       "        cursor = 'crosshair';\n",
       "        break;\n",
       "    case 3:\n",
       "        cursor = 'move';\n",
       "        break;\n",
       "    }\n",
       "    fig.rubberband_canvas.style.cursor = cursor;\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.handle_message = function(fig, msg) {\n",
       "    fig.message.textContent = msg['message'];\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.handle_draw = function(fig, msg) {\n",
       "    // Request the server to send over a new figure.\n",
       "    fig.send_draw_message();\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.handle_image_mode = function(fig, msg) {\n",
       "    fig.image_mode = msg['mode'];\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.updated_canvas_event = function() {\n",
       "    // Called whenever the canvas gets updated.\n",
       "    this.send_message(\"ack\", {});\n",
       "}\n",
       "\n",
       "// A function to construct a web socket function for onmessage handling.\n",
       "// Called in the figure constructor.\n",
       "mpl.figure.prototype._make_on_message_function = function(fig) {\n",
       "    return function socket_on_message(evt) {\n",
       "        if (evt.data instanceof Blob) {\n",
       "            /* FIXME: We get \"Resource interpreted as Image but\n",
       "             * transferred with MIME type text/plain:\" errors on\n",
       "             * Chrome.  But how to set the MIME type?  It doesn't seem\n",
       "             * to be part of the websocket stream */\n",
       "            evt.data.type = \"image/png\";\n",
       "\n",
       "            /* Free the memory for the previous frames */\n",
       "            if (fig.imageObj.src) {\n",
       "                (window.URL || window.webkitURL).revokeObjectURL(\n",
       "                    fig.imageObj.src);\n",
       "            }\n",
       "\n",
       "            fig.imageObj.src = (window.URL || window.webkitURL).createObjectURL(\n",
       "                evt.data);\n",
       "            fig.updated_canvas_event();\n",
       "            fig.waiting = false;\n",
       "            return;\n",
       "        }\n",
       "        else if (typeof evt.data === 'string' && evt.data.slice(0, 21) == \"data:image/png;base64\") {\n",
       "            fig.imageObj.src = evt.data;\n",
       "            fig.updated_canvas_event();\n",
       "            fig.waiting = false;\n",
       "            return;\n",
       "        }\n",
       "\n",
       "        var msg = JSON.parse(evt.data);\n",
       "        var msg_type = msg['type'];\n",
       "\n",
       "        // Call the  \"handle_{type}\" callback, which takes\n",
       "        // the figure and JSON message as its only arguments.\n",
       "        try {\n",
       "            var callback = fig[\"handle_\" + msg_type];\n",
       "        } catch (e) {\n",
       "            console.log(\"No handler for the '\" + msg_type + \"' message type: \", msg);\n",
       "            return;\n",
       "        }\n",
       "\n",
       "        if (callback) {\n",
       "            try {\n",
       "                // console.log(\"Handling '\" + msg_type + \"' message: \", msg);\n",
       "                callback(fig, msg);\n",
       "            } catch (e) {\n",
       "                console.log(\"Exception inside the 'handler_\" + msg_type + \"' callback:\", e, e.stack, msg);\n",
       "            }\n",
       "        }\n",
       "    };\n",
       "}\n",
       "\n",
       "// from http://stackoverflow.com/questions/1114465/getting-mouse-location-in-canvas\n",
       "mpl.findpos = function(e) {\n",
       "    //this section is from http://www.quirksmode.org/js/events_properties.html\n",
       "    var targ;\n",
       "    if (!e)\n",
       "        e = window.event;\n",
       "    if (e.target)\n",
       "        targ = e.target;\n",
       "    else if (e.srcElement)\n",
       "        targ = e.srcElement;\n",
       "    if (targ.nodeType == 3) // defeat Safari bug\n",
       "        targ = targ.parentNode;\n",
       "\n",
       "    // jQuery normalizes the pageX and pageY\n",
       "    // pageX,Y are the mouse positions relative to the document\n",
       "    // offset() returns the position of the element relative to the document\n",
       "    var x = e.pageX - $(targ).offset().left;\n",
       "    var y = e.pageY - $(targ).offset().top;\n",
       "\n",
       "    return {\"x\": x, \"y\": y};\n",
       "};\n",
       "\n",
       "/*\n",
       " * return a copy of an object with only non-object keys\n",
       " * we need this to avoid circular references\n",
       " * http://stackoverflow.com/a/24161582/3208463\n",
       " */\n",
       "function simpleKeys (original) {\n",
       "  return Object.keys(original).reduce(function (obj, key) {\n",
       "    if (typeof original[key] !== 'object')\n",
       "        obj[key] = original[key]\n",
       "    return obj;\n",
       "  }, {});\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.mouse_event = function(event, name) {\n",
       "    var canvas_pos = mpl.findpos(event)\n",
       "\n",
       "    if (name === 'button_press')\n",
       "    {\n",
       "        this.canvas.focus();\n",
       "        this.canvas_div.focus();\n",
       "    }\n",
       "\n",
       "    var x = canvas_pos.x * mpl.ratio;\n",
       "    var y = canvas_pos.y * mpl.ratio;\n",
       "\n",
       "    this.send_message(name, {x: x, y: y, button: event.button,\n",
       "                             step: event.step,\n",
       "                             guiEvent: simpleKeys(event)});\n",
       "\n",
       "    /* This prevents the web browser from automatically changing to\n",
       "     * the text insertion cursor when the button is pressed.  We want\n",
       "     * to control all of the cursor setting manually through the\n",
       "     * 'cursor' event from matplotlib */\n",
       "    event.preventDefault();\n",
       "    return false;\n",
       "}\n",
       "\n",
       "mpl.figure.prototype._key_event_extra = function(event, name) {\n",
       "    // Handle any extra behaviour associated with a key event\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.key_event = function(event, name) {\n",
       "\n",
       "    // Prevent repeat events\n",
       "    if (name == 'key_press')\n",
       "    {\n",
       "        if (event.which === this._key)\n",
       "            return;\n",
       "        else\n",
       "            this._key = event.which;\n",
       "    }\n",
       "    if (name == 'key_release')\n",
       "        this._key = null;\n",
       "\n",
       "    var value = '';\n",
       "    if (event.ctrlKey && event.which != 17)\n",
       "        value += \"ctrl+\";\n",
       "    if (event.altKey && event.which != 18)\n",
       "        value += \"alt+\";\n",
       "    if (event.shiftKey && event.which != 16)\n",
       "        value += \"shift+\";\n",
       "\n",
       "    value += 'k';\n",
       "    value += event.which.toString();\n",
       "\n",
       "    this._key_event_extra(event, name);\n",
       "\n",
       "    this.send_message(name, {key: value,\n",
       "                             guiEvent: simpleKeys(event)});\n",
       "    return false;\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.toolbar_button_onclick = function(name) {\n",
       "    if (name == 'download') {\n",
       "        this.handle_save(this, null);\n",
       "    } else {\n",
       "        this.send_message(\"toolbar_button\", {name: name});\n",
       "    }\n",
       "};\n",
       "\n",
       "mpl.figure.prototype.toolbar_button_onmouseover = function(tooltip) {\n",
       "    this.message.textContent = tooltip;\n",
       "};\n",
       "mpl.toolbar_items = [[\"Home\", \"Reset original view\", \"fa fa-home icon-home\", \"home\"], [\"Back\", \"Back to  previous view\", \"fa fa-arrow-left icon-arrow-left\", \"back\"], [\"Forward\", \"Forward to next view\", \"fa fa-arrow-right icon-arrow-right\", \"forward\"], [\"\", \"\", \"\", \"\"], [\"Pan\", \"Pan axes with left mouse, zoom with right\", \"fa fa-arrows icon-move\", \"pan\"], [\"Zoom\", \"Zoom to rectangle\", \"fa fa-square-o icon-check-empty\", \"zoom\"], [\"\", \"\", \"\", \"\"], [\"Download\", \"Download plot\", \"fa fa-floppy-o icon-save\", \"download\"]];\n",
       "\n",
       "mpl.extensions = [\"eps\", \"jpeg\", \"pdf\", \"png\", \"ps\", \"raw\", \"svg\", \"tif\"];\n",
       "\n",
       "mpl.default_extension = \"png\";var comm_websocket_adapter = function(comm) {\n",
       "    // Create a \"websocket\"-like object which calls the given IPython comm\n",
       "    // object with the appropriate methods. Currently this is a non binary\n",
       "    // socket, so there is still some room for performance tuning.\n",
       "    var ws = {};\n",
       "\n",
       "    ws.close = function() {\n",
       "        comm.close()\n",
       "    };\n",
       "    ws.send = function(m) {\n",
       "        //console.log('sending', m);\n",
       "        comm.send(m);\n",
       "    };\n",
       "    // Register the callback with on_msg.\n",
       "    comm.on_msg(function(msg) {\n",
       "        //console.log('receiving', msg['content']['data'], msg);\n",
       "        // Pass the mpl event to the overriden (by mpl) onmessage function.\n",
       "        ws.onmessage(msg['content']['data'])\n",
       "    });\n",
       "    return ws;\n",
       "}\n",
       "\n",
       "mpl.mpl_figure_comm = function(comm, msg) {\n",
       "    // This is the function which gets called when the mpl process\n",
       "    // starts-up an IPython Comm through the \"matplotlib\" channel.\n",
       "\n",
       "    var id = msg.content.data.id;\n",
       "    // Get hold of the div created by the display call when the Comm\n",
       "    // socket was opened in Python.\n",
       "    var element = $(\"#\" + id);\n",
       "    var ws_proxy = comm_websocket_adapter(comm)\n",
       "\n",
       "    function ondownload(figure, format) {\n",
       "        window.open(figure.imageObj.src);\n",
       "    }\n",
       "\n",
       "    var fig = new mpl.figure(id, ws_proxy,\n",
       "                           ondownload,\n",
       "                           element.get(0));\n",
       "\n",
       "    // Call onopen now - mpl needs it, as it is assuming we've passed it a real\n",
       "    // web socket which is closed, not our websocket->open comm proxy.\n",
       "    ws_proxy.onopen();\n",
       "\n",
       "    fig.parent_element = element.get(0);\n",
       "    fig.cell_info = mpl.find_output_cell(\"<div id='\" + id + \"'></div>\");\n",
       "    if (!fig.cell_info) {\n",
       "        console.error(\"Failed to find cell for figure\", id, fig);\n",
       "        return;\n",
       "    }\n",
       "\n",
       "    var output_index = fig.cell_info[2]\n",
       "    var cell = fig.cell_info[0];\n",
       "\n",
       "};\n",
       "\n",
       "mpl.figure.prototype.handle_close = function(fig, msg) {\n",
       "    var width = fig.canvas.width/mpl.ratio\n",
       "    fig.root.unbind('remove')\n",
       "\n",
       "    // Update the output cell to use the data from the current canvas.\n",
       "    fig.push_to_output();\n",
       "    var dataURL = fig.canvas.toDataURL();\n",
       "    // Re-enable the keyboard manager in IPython - without this line, in FF,\n",
       "    // the notebook keyboard shortcuts fail.\n",
       "    IPython.keyboard_manager.enable()\n",
       "    $(fig.parent_element).html('<img src=\"' + dataURL + '\" width=\"' + width + '\">');\n",
       "    fig.close_ws(fig, msg);\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.close_ws = function(fig, msg){\n",
       "    fig.send_message('closing', msg);\n",
       "    // fig.ws.close()\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.push_to_output = function(remove_interactive) {\n",
       "    // Turn the data on the canvas into data in the output cell.\n",
       "    var width = this.canvas.width/mpl.ratio\n",
       "    var dataURL = this.canvas.toDataURL();\n",
       "    this.cell_info[1]['text/html'] = '<img src=\"' + dataURL + '\" width=\"' + width + '\">';\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.updated_canvas_event = function() {\n",
       "    // Tell IPython that the notebook contents must change.\n",
       "    IPython.notebook.set_dirty(true);\n",
       "    this.send_message(\"ack\", {});\n",
       "    var fig = this;\n",
       "    // Wait a second, then push the new image to the DOM so\n",
       "    // that it is saved nicely (might be nice to debounce this).\n",
       "    setTimeout(function () { fig.push_to_output() }, 1000);\n",
       "}\n",
       "\n",
       "mpl.figure.prototype._init_toolbar = function() {\n",
       "    var fig = this;\n",
       "\n",
       "    var nav_element = $('<div/>')\n",
       "    nav_element.attr('style', 'width: 100%');\n",
       "    this.root.append(nav_element);\n",
       "\n",
       "    // Define a callback function for later on.\n",
       "    function toolbar_event(event) {\n",
       "        return fig.toolbar_button_onclick(event['data']);\n",
       "    }\n",
       "    function toolbar_mouse_event(event) {\n",
       "        return fig.toolbar_button_onmouseover(event['data']);\n",
       "    }\n",
       "\n",
       "    for(var toolbar_ind in mpl.toolbar_items){\n",
       "        var name = mpl.toolbar_items[toolbar_ind][0];\n",
       "        var tooltip = mpl.toolbar_items[toolbar_ind][1];\n",
       "        var image = mpl.toolbar_items[toolbar_ind][2];\n",
       "        var method_name = mpl.toolbar_items[toolbar_ind][3];\n",
       "\n",
       "        if (!name) { continue; };\n",
       "\n",
       "        var button = $('<button class=\"btn btn-default\" href=\"#\" title=\"' + name + '\"><i class=\"fa ' + image + ' fa-lg\"></i></button>');\n",
       "        button.click(method_name, toolbar_event);\n",
       "        button.mouseover(tooltip, toolbar_mouse_event);\n",
       "        nav_element.append(button);\n",
       "    }\n",
       "\n",
       "    // Add the status bar.\n",
       "    var status_bar = $('<span class=\"mpl-message\" style=\"text-align:right; float: right;\"/>');\n",
       "    nav_element.append(status_bar);\n",
       "    this.message = status_bar[0];\n",
       "\n",
       "    // Add the close button to the window.\n",
       "    var buttongrp = $('<div class=\"btn-group inline pull-right\"></div>');\n",
       "    var button = $('<button class=\"btn btn-mini btn-primary\" href=\"#\" title=\"Stop Interaction\"><i class=\"fa fa-power-off icon-remove icon-large\"></i></button>');\n",
       "    button.click(function (evt) { fig.handle_close(fig, {}); } );\n",
       "    button.mouseover('Stop Interaction', toolbar_mouse_event);\n",
       "    buttongrp.append(button);\n",
       "    var titlebar = this.root.find($('.ui-dialog-titlebar'));\n",
       "    titlebar.prepend(buttongrp);\n",
       "}\n",
       "\n",
       "mpl.figure.prototype._root_extra_style = function(el){\n",
       "    var fig = this\n",
       "    el.on(\"remove\", function(){\n",
       "\tfig.close_ws(fig, {});\n",
       "    });\n",
       "}\n",
       "\n",
       "mpl.figure.prototype._canvas_extra_style = function(el){\n",
       "    // this is important to make the div 'focusable\n",
       "    el.attr('tabindex', 0)\n",
       "    // reach out to IPython and tell the keyboard manager to turn it's self\n",
       "    // off when our div gets focus\n",
       "\n",
       "    // location in version 3\n",
       "    if (IPython.notebook.keyboard_manager) {\n",
       "        IPython.notebook.keyboard_manager.register_events(el);\n",
       "    }\n",
       "    else {\n",
       "        // location in version 2\n",
       "        IPython.keyboard_manager.register_events(el);\n",
       "    }\n",
       "\n",
       "}\n",
       "\n",
       "mpl.figure.prototype._key_event_extra = function(event, name) {\n",
       "    var manager = IPython.notebook.keyboard_manager;\n",
       "    if (!manager)\n",
       "        manager = IPython.keyboard_manager;\n",
       "\n",
       "    // Check for shift+enter\n",
       "    if (event.shiftKey && event.which == 13) {\n",
       "        this.canvas_div.blur();\n",
       "        event.shiftKey = false;\n",
       "        // Send a \"J\" for go to next cell\n",
       "        event.which = 74;\n",
       "        event.keyCode = 74;\n",
       "        manager.command_mode();\n",
       "        manager.handle_keydown(event);\n",
       "    }\n",
       "}\n",
       "\n",
       "mpl.figure.prototype.handle_save = function(fig, msg) {\n",
       "    fig.ondownload(fig, null);\n",
       "}\n",
       "\n",
       "\n",
       "mpl.find_output_cell = function(html_output) {\n",
       "    // Return the cell and output element which can be found *uniquely* in the notebook.\n",
       "    // Note - this is a bit hacky, but it is done because the \"notebook_saving.Notebook\"\n",
       "    // IPython event is triggered only after the cells have been serialised, which for\n",
       "    // our purposes (turning an active figure into a static one), is too late.\n",
       "    var cells = IPython.notebook.get_cells();\n",
       "    var ncells = cells.length;\n",
       "    for (var i=0; i<ncells; i++) {\n",
       "        var cell = cells[i];\n",
       "        if (cell.cell_type === 'code'){\n",
       "            for (var j=0; j<cell.output_area.outputs.length; j++) {\n",
       "                var data = cell.output_area.outputs[j];\n",
       "                if (data.data) {\n",
       "                    // IPython >= 3 moved mimebundle to data attribute of output\n",
       "                    data = data.data;\n",
       "                }\n",
       "                if (data['text/html'] == html_output) {\n",
       "                    return [cell, data, j];\n",
       "                }\n",
       "            }\n",
       "        }\n",
       "    }\n",
       "}\n",
       "\n",
       "// Register the function which deals with the matplotlib target/channel.\n",
       "// The kernel may be null if the page has been refreshed.\n",
       "if (IPython.notebook.kernel != null) {\n",
       "    IPython.notebook.kernel.comm_manager.register_target('matplotlib', mpl.mpl_figure_comm);\n",
       "}\n"
      ],
      "text/plain": [
       "<IPython.core.display.Javascript object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "data": {
      "text/html": [
       "<img src=\"\" width=\"639.4\">"
      ],
      "text/plain": [
       "<IPython.core.display.HTML object>"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    }
   ],
   "source": [
    "from mpl_toolkits.mplot3d import Axes3D\n",
    "import matplotlib.pyplot as plt\n",
    "%matplotlib notebook\n",
    "fig = plt.figure()\n",
    "ax = fig.add_subplot(111, projection='3d')\n",
    "ax.scatter(test_z[:,1],test_z[:,0], test_z[:,2], c=test_labels.astype(int))\n",
    "ax.set_xlabel('Encoded')\n",
    "ax.set_ylabel('Euclidean')\n",
    "ax.set_zlabel('Cosine')\n",
    "plt.show()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.6.4"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}
